A few methods follow:
1. Use non-DMA links.
2. Use trusted hardware/firmware that mediates things properly.
3. Use IOMMU to enforce access controls on DMA.
4. Use a combo of full safety in system and careful API for access to DMA features.
I used 1 and 2. A few use 4. Number 3 is most common with basic version mainstreaming. Not enough, though, as complex firmware and OS's still provide attack opportunities.
Note: There's also interrupt floods and other esoteric issues to counter. So, it's the start rather than full solution. EMSEC issues too with malicious peripherals.
Many issues due to how things connect and store information as burfog said. There's not just one thing. Even my microkernel recommendation is only for the start of the software part of a secure system. Actually, trustworthy CPU, ROM, bootloader and drivers are the start if we're being technical. ;)
"Most systems are unable to keep DMA from scribbling all over everything in RAM, and probably all are unable to keep it from scribbling all over a filesystem."
In what context do you mean? When would this occur?
Thanks