I wonder if you could use something like the RetroArch rollback solution[1] - render multiple versions of the stream, then pick the closest one based on player input. With a fast enough Internet connection, you could let the _client_ pick, then there is no lag.

Of course, the hardware you have to throw at it to do that quickly becomes ridiculous, unless you can come up with a way of efficiently rendering the multiple versions.

[1] https://www.libretro.com/index.php/retroarch-1-7-2%E2%80%8A-...