Replacing WebRTC: real-time latency with WebTransport and WebCodecs

Re:Replacing WebRTC: real-time latency with WebTransport and... by Rodeoclash on Monday October 30, 2023 @04:59PM (#38075737)

Wow, great to see some work in this space. I've been wanting to do reverse playback, frame accurate seek and step by step forward and back rendering in the browser for esports game analysis. The regular video tag gets you somewhat of the way there but navigating frame by frame will sometimes jump an extra frame. Likewise trying to stop at an exact point will often be 1 or 2 frames off where you should be. Firefox is much worse, when pausing at a time you could +-12 frames where you should be.

I must find some time to dig into this, thanks for sharing it.

[ Parent | on HN ]

Re:Replacing WebRTC: real-time latency with WebTransport and... by jampekka on Monday October 30, 2023 @06:16PM (#38076712)

I have it working with WebCodecs, but currently i-frames only videos and all the decoded frames are read to memory. Not impossible to lift these restrictions, but the current WebCodec API will likely make it a bit brittle (and/or janky). For my current case this is not a big problem so I haven't fought with it too much.

Figuring out libav.js demuxing may be a bit of a challenge, even though the API is quite nice as traditional AV APIs go. I'll put out my small wrapper for these in a few days.

Edit: to be clear I don't have anything to do with libav.js other than happening to find it and using it to scratch my itch. Most demuxing examples for WebCodecs use mp4box.js which really makes one a bit uncomfortably intimate with guts of the MP4 format.

[ Parent | on HN ]