A forum for reverse engineering, OS internals and malware analysis 

Discussion on reverse-engineering and debugging.
 #32879  by Iradicator
 Tue Apr 30, 2019 12:57 pm

I want to build some sort of API monitor by hooking every ntdll function. So each hooked function will call the original implementation, and than add this call to the std based data structure.

However, I encountered a scenario of recursive hooking where my hooked function is indirectly calling itself.

Therefore, I tried to use the Tls memory to set a bit per thread that indicate that current function is called from hooked function, so avoid calling the hook again (execute the original function only).

But my recursive hook guard also indirectly triggers calls for memory allocation functions like NtAllocateVirtualMemory, and therefore i'm currently avoid hooking those functions.

Perhaps anybody has encountered a similar issue and implemented hook reentrancy guard in a way that doesn't triggered any memory allocation function (which might be imposible since even if you call a new function and your stack
memory is insufficient, it should allocate more memory).