If changing all the HW we use was as easy as waving a wand, then your argument would be sound. Unfortunately, it’s super hard to get folks to use different HW. And because of how silicon economics work out, a new upstart architecture with limited users is sure to experience slower perf scaling than the mainstream HW. That strongly disincentivizes anyone from being an early adopter of exotic new HW. I think that’s why CHERI isn’t mainstream yet despite a decade of investment.
Hence why even ignoring Fil-C, it’s probably more realistic to rewrite stuff in Rust than it is to switch to CHERI. Fil-C adds a third option where you pay some perf but you don’t have to rewrite your code or switch what HW you use. CHERI also costs perf - practically speaking, available CHERI HW is more than 6x slower than the HW Fil-C can run on. That’s fundamental, due to silicon economics.
Finally, I think you’re overselling the CHERI defense in depth. My understanding is that CHERIoT doesn’t use virtual memory and that there are subtle reasons why virtual memory and CHERI put together gets weird, especially if you want to support capability revocation. On the other hand, Fil-C works on top of virtual memory so you get both virtual memory protection and capability protection, hence more defense in depth.
One of the advantages of CHERI is that you can get a safe system with no virtual memory, no overheads of TLB lookups and hardware, etc. but you don't have to do that. The CHERI arm and riscv64 designs have the normal memory management hardware.
No virtual memory is not an advantage of CHERI, it’s a disadvantage. It means CHERI replaces a very well understood security barrier (virtual memory) with one that is less well understood (no matter how many proofs you write about it). I would be more willing to buy CHERI’s claims if they had kept virtual memory. Fil-C plays nice with virtual memory so there’s no need to disable it.
For unsafe Rust code to be sound, it has to uphold the invariant that no location pointed to by an &mut is modified through a pointer not derived from that &mut. However, there's no such restriction on *mut pointers ("raw pointers"), and it is in many cases perfectly sound (but unsafe, of course) to cast from *mut to &mut, or to directly mutate through a *mut. Without maintaining global information about what references are live, it sounds really, really hard to guarantee this.
It's still useful for making swapping work efficiently, but that's not relevant on a tiny embedded device.
Fil-C just says that shared memory mappings are integer-only, so trying to place a capability there instantly traps. That’s both sound and adequate even for sophisticated uses of shared memory.
My understanding is that this is a conundrum for CHERI, but maybe my understanding is wrong.
To put it another way why not remove existing hardware security abstractions we do have? Ditch virtualization (or hugely strip it back), get rid of different privilege levels, loose page-table enforced permissions. After all you can just build safe software why bother with these things?
I for one don't think this is a sensible line of argument, defence in depth is a good idea, build your software to be as secure as possible then build your hardware such that if there is some flaw in your software's defences you can keep things contained though hardware abstractions.
Even with improved languages there's a giant body of existing code you need to work with, with compartmentalization in CHERI you can keep this stuff safely contained. Plus how do you guarantee everything running on the system has been built in your safe language and compiled with your blessed known good compiler? Feasible in some places (like embedded systems) far less feasible in others.