Hence my enthusiasm :-)

What level of interposition can one do with these hard cpu cores? Are the cache eviction policies pluggable? Is "user code" in the path between the cpu and main memory?

On the Ultrascale Zynq's you can build RAM that is attached to the fabric and available to the processor across the AMBA bus. It isn't something I personally have played with a whole bunch so I can't give you a better answer then to look at the Zynq data sheets.

I do know you can make user hardware "appear" as memory to the main CPU.

Neat. I am stoked to play with the new PolarFire RISC-V Icicle board.

https://www.microsemi.com/existing-parts/parts/152514

https://www.crowdsupply.com/microchip/polarfire-soc-icicle-k...