I would guess the processor would just cause an exception/interrupt and it would just call an OS level exception/interrupt handler which would probably tell the user what the exception/interrupt was that occured; in this case an unsupported instruction.