You start with a problem, then you think what system might be interacting with that particular subsystem. Then you start to poke around in there trying to intercept, dump code, add debugging and so on.

But are you poking around macOS? If it’s totally unique hardware unlike anything else, what would you even poke around?

Side channels like timing during different tests is one possibility. You can also reverse the operating system to understand more about how it may be handling interrupts or unique hardware. Then there's also dumping any onboard firmware memory to learn about the boot process. With some creativity, there's loads of ways to infer functionality of the processor/system enough to develop over it.