A forum for reverse engineering, OS internals and malware analysis 

Forum for discussion about user-mode development.
 #8722  by _Lynn
 Sat Sep 24, 2011 4:47 am
Hey guys, hopefully a simple question here. Was wondering about running a trace over sysenter calls, I noticed more often then not while having the trap flag set before and then after a syscall returns, it usually generates an access violation when returning to the caller.

What is the reasoning for this?

Also, what is an efficient approach to handling this, best I could come up with was just checking ahead every instruction, then setting an int3 after the call. Only problem is, for native routines on another thread could hit this breakpoint.

First and foremost though, is keeping the trap flag set over a sysenter known to cause an access violation or am I doing something else wrong.

Thanks! ;)
 #9004  by kmd
 Fri Oct 07, 2011 3:10 am
if TF flag is set and sysenter used -- exception handler will be called, it lookups if EIP is in KiFastCallEntry and if true unsets TF and continues to execute service because fast call do not use stack