Unfortunately the ability to playback videos via MSE API with low latency seems to vary quite a bit from device to device. You could also try an older release (v5.*). Those work by repeatedly sending PNGs and updating an image tag, actually the gif in the Readme is from that version. If this works better for you I will probably add it back as fallback in case video playback is to slow.
I wonder if you could integrate this HTML5 VNC client: https://github.com/novnc/noVNC , since the complicated stuff above will be taken care of by the VNC server.
I feel like the boss in Office Space saying "if you could just go ahead and...".
This is a possibility but encoding is already pretty optimized (ffmpeg and libx264 with ultrafast as preset), which is faster than my previous approach of just streaming PNGs. And streaming PNGs was not that slow either, even on a rather old laptop. That's why I suspect there's something wrong on the tablet.
Regarding noVNC: What latencies do they achieve? I am not a webdev so I guess they might have some more tricks up their sleeves, I will probably have a look. As far as I can tell noVNC does not support things like multitouch and pressure sensitivity so that rules out just using it (at least for my usecase).
If someone wants to build a mobile app I am more than happy to provide other means of connection to Weylus than websockets. This also might bring the latency down as udp streaming and more control over video decoding and playing is possible then.
It looks like this is more of a mirroring setup where you can use the tablet for touch-enabled input.
Regardless, this project looks incredibly cool. The cross-OS support is a killer feature.
Anyway, got a Luna Display and couldn't be happier.
As far as I can tell this is not possible: https://tools.ietf.org/html/rfc6143#section-7.5.5
Also selecting a specific window, which is possible in Weylus, does not seem to work with vnc.
It is written mostly in Rust, some C and a tiny bit Typescript and works by serving a webpage that uses the browser API to capture pen, touch and mouse events from the tablet. Additionally the desktop or a selected window is captured and encoded to a video stream via ffmpeg. This stream is played in the browser using the Media Source Extensions API.
This should also run on macOS and Windows but with very limited features.