What an overlay does is hook the 'present' function, does more drawing, then calls the base 'present' function. It's easy enough to detect when a draw call happens during a call to 'present', that tells you that an overlay is active.

Then you're faced with the problem of how deep your hook is, you either want it to run early (before other hooks), or late (after other hooks but before actual function). When you're trying to capture the overlay, you want your hook to run late.

The cop-out is to hook D3DKMTPresent instead and hope nobody else hooked it.