Some more recent designs seem to be standard µATX with standard PCIe, though, so these should be possible to use with virtualisation and the mandatory IOMMU for the proprietary hardware (Feature locks / DRM might be cranky, tho).
And before the mainstream OS era these were usually 100 % custom jobs, completely unlike a PC. 68k, MIPS, PPC, and even proprietary designs were used.
I do remember some sort of extension that could install drivers that made it behave like a real time OS. However at that point I think seperating the control out to seperate hardware with a real time OS seems like the obvious choice, and then keeping the interface on a genereal purpose PC. Though it wouldn't be the first time a vendor made a stupid choice.
but, they said, "don't worry, we'll keep supporting XP!"
...right