If the extra instructions also operate in extra state (e.g. extra registers) a kernel needs to know about their existence so it can correctly save and restore state on context switches

Not necessarily/really, the custom extensions still need to be enabled by the kernel before they can be used.

As such, it isn’t actually an issue.

You're confusing MSRs (which don't have to be saved/restored on context switch) with GPRs (which do).

Well if there is a truly undocumented extension, how do I know it doesn't come with it's own registers (e.g. like a floating point unit does)

Apple reserves certain MSRs that apply on a per-process basis, and thus must be applied on context switch.

It doesn’t need to use them, but it must be aware of them, insofar as they may introduce security problems.

As an example, if the kernel doesn’t know of DMA channels, and it requires setup code to prevent user-level code from using them to copy across process boundaries, the kernel will run fine, but have glaring security problems.

What dma channels doesn't require mapping registers into the user space process to work? There aren't usually magic instructions you have to opt into disabling as far as I know.

Those are not security problems, they are insecurity features readjusts tinfoil.