Page 1 of 1

Prevent recursive hooking when using detours

PostPosted:Tue Apr 30, 2019 12:57 pm
by Iradicator
Hi,

I want to build some sort of API monitor by hooking every ntdll function. So each hooked function will call the original implementation, and than add this call to the std based data structure.

However, I encountered a scenario of recursive hooking where my hooked function is indirectly calling itself.

Therefore, I tried to use the Tls memory to set a bit per thread that indicate that current function is called from hooked function, so avoid calling the hook again (execute the original function only).

But my recursive hook guard also indirectly triggers calls for memory allocation functions like NtAllocateVirtualMemory, and therefore i'm currently avoid hooking those functions.

Perhaps anybody has encountered a similar issue and implemented hook reentrancy guard in a way that doesn't triggered any memory allocation function (which might be imposible since even if you call a new function and your stack
memory is insufficient, it should allocate more memory).

thanks

Re: Prevent recursive hooking when using detours

PostPosted:Sun Aug 18, 2019 12:32 pm
by Siro
You mean that inside the callback you have
Code: Select all
void MarkInsideMyHook(unsigned int Hook)
{
    auto ptr = TlsGetValue(g_tlsIndex);
    if (ptr == nullptr)
        ptr = LocalAlloc(....)  // <--- this calls NtAllocateVirtualMemory ?
     ...
}

You could build a static solution (which has its own drawbacks).
If you assume you handle at most MAX threads at once, then you can pre-allocate a global hashset on ATTACH_PROCESS, and use `idx = hash(tid) ` where you insert the relevant tid on first entry, and remove it when you leave.

Then you check the same hashtable for presense of tid.

If you have a present MAX size , you can avoid using allocations, and even work safely by using interlocked functions to set/get the tid inside the hashmap.