It sounds like you want BokehJS. It was one of the alternatives I was recommended while I was exploring, but for various reasons my particular use case is not so easy to integrate (plus my backend was already in Rust).

https://github.com/bokeh/bokeh

I did do a basic test, and the raw rects-on-screen performance is roughly comparable to my final solution.