I was working on exactly this (though not to Lisp, per se) before I got distracted and COVID came. I was calling it "Retro-V". In Verilog, I had a RISC-V core hooked up to my own home-built video logic, and was working on the SD card controller before I shelved it.

In the meantime various RISC-V SoCs have come to market which could probably do the job well.