Detours rewrites instructions at the start of the detoured function. Why should it know or care whether the instructions it's overwriting happen to be ones written by a previous detour? Why would you need to synchronize?

Doesn't the Microsoft implementation work by having nop instruction as the first instruction of the function, which can then be patched to a jump to the patched code, that finally jumps back to the original function past the NOP?

Naive implementation of this scheme doesn't allow stacking.

If you have the standard two-byte backwards jump pointing to a longer jump instruction, you also have another happy case for overriding it (just change the long jump).

Meanwhile, the Microsoft Detours library supports detouring other kinds of instructions at the beginning of a function. It's not exclusively for the 'two byte nop and blank space' thing.

[delayed]

If you use OBS, a game capture works the same way, it hooks the DirectX (or whatever API) functions to grab the rendered scene directly. It often has issues picking up overlays because there isn't really any synchronisation, just whatever made the last detour fires first. Like the sibling commit says, you can stack detours, it's only an issue for OBS if something that runs after it renders more things to the screen.

There is a setting to make it pick up overlays, which works reasonably well, I'm not entirely sure how it's implemented.

What an overlay does is hook the 'present' function, does more drawing, then calls the base 'present' function. It's easy enough to detect when a draw call happens during a call to 'present', that tells you that an overlay is active.

Then you're faced with the problem of how deep your hook is, you either want it to run early (before other hooks), or late (after other hooks but before actual function). When you're trying to capture the overlay, you want your hook to run late.

The cop-out is to hook D3DKMTPresent instead and hope nobody else hooked it.