That works as a heuristic, but it's not perfect, since JITs and self-modifying code are a thing.
I expect the chip will raise a fault and the OS will move the process.
[0] https://www.neotextus.net/hpca12.pdf
(I think the "3.2 QuickIA Software Support" section is interesting, if nearly a decade old by now)
Runtime detection would be the only actual concern here, but you can easily just advertise the common baseline. As in, just pretend the sunny cove core doesn't support AVX512. The only problem then becomes the big core is potentially slower than it could be, but given how rare things like AVX512 is in typical desktop applications will anyone actually care?
EDIT: Oh, and this appears to be what they're basically doing. Even though Sunny Cove itself supports AVX-512, it's being disabled in this application: "One thing we can confirm in advance – the Sunny Cove does not appear to be AVX-512 enabled."
The program machine code can be scanned when loaded to look for specific assembly opcodes to determine the required capability of the CPU to execute it on. The code with instructions not fitted for Atom will be sent to the main CPU only.
Edit: Just a thought. The OS can install an illegal-opcode exception handler. When a process is first run on the small CPU, the unsupported opcode will raise the exception. The exception handler can simply set the processor affinity of the process to the main CPU and put it to sleep. The OS will handle it like it normally would - putting the process in the run queue of the affined processor.