While it's an interesting idea, you would need one or even two of the big three players (MS, Apple, Google) to push this onto the market, otherwise you'd have a complete driver desert.
I'd go one level above: make a HIL bridge between standard x86 drivers and your lisp machine code.
Well you could also have machine code be a lisp (as did many lisp machines, which used lisps from the microcode up).