The scheduler already knows if you've used AVX or SSE instructions. You don't think they waste time backing up all those registers when you've never used them do you?

You can set a flag causing an exception when an FPU instruction is used, and handle it by setting a flag saying you need to backup those registers.

You can also handle Undefined Operation exceptions by moving the thread to a core that can handle the specific instruction.