DaleCurtis

FWIW, the getUserMedia() portion of such a setup remains the same, so you don't lose AEC or anything else coupled there.

Ah, I didn't realize that always happened. I thought it was only if you did something that might have OS specific rendering characteristics (text-draws, etc).

Maybe having an ImageEncoder API might be worthwhile after all then https://github.com/w3c/webcodecs/issues/204.

FWIW, you can do this with a few lines of JS in the browser using canvas.drawImage() from an img element followed by canvas.toBlob().

Thanks for writing this! It skips my favorite edgy product name though, the X-Fi Fatal1ty cards: https://en.wikipedia.org/wiki/Sound_Blaster_X-Fi :)

If only that were true, battery usage would be much better :) Just consider the prominence of content like tiktoks/shorts/reels/etc alone.

Unfortunately canvas (rgb'ish) can't overlay as efficiently as <video> (yuv'ish), so there is some power cost relative to the lowest power video overlays.

It really only matters in long form content where nothing else on the page is changing though.

The code in Chromium which handles this suspension is here: https://source.chromium.org/chromium/chromium/src/+/main:med...

Basically after detecting silence for 30 seconds or so it switches from a sink backed by the OS audio device to a null sink.

Note: Since this uses a different clock than the audio device we have received some reports that when the context is finally used there can be some distortion at specific tones. The workaround is for sites to use the suspend resume API mentioned in the article.

Chrome supports HEVC with transparency on macOS since VideoToolbox does. Other platform decoders don't support it though.

What a fun excursion :) You can also use the ImageDecoder API: https://developer.mozilla.org/en-US/docs/Web/API/ImageDecode... and VideoFrame.copyTo: https://developer.mozilla.org/en-US/docs/Web/API/VideoFrame/... to skip canvas entirely.

I still remembered my id from all those years ago, 1569200. I was excited to read others were logging in with their old numbers, so I tried the password I thought I had used, but no luck.

Feeling adventurous, I dug through my archives and found my old ICQ database and followed https://sec.sipsik.net/tuts/net/icq.txt and used the code at https://web.archive.org/web/20070209002044/https://rejetto.c... to decrypt my password...

Sadly it was the one I had tried :( It was a fun trip through nostalgia at least!

IIRC, that bot used to be called bugdroid. I forget when it switched over, probably somewhere in 2020.

Very cool! I think it's missing some entries though. I'm pretty sure we've had at least one in third_party/ffmpeg. Those fixes often land upstream first which might make tracking difficult.

Even if you have crash reporting disabled there should be a .dmp generated somewhere in the user profile directory. Manually uploading that to a bug at https://crbug.com/new would allow a Chrome developer to debug it.

If you can't share the dump for similar reasons to why you have crash reporting disabled, you can build minidump_stackwalk from Chromium and use it to generate an unsymbolized stack trace that you can post to the bug. A Chrome developer can then symbolize it.

https://www.chromium.org/developers/decoding-crash-dumps/ has some more details.

Often Chrome doesn't know when more frames are needed either, so it's not something we could add an API for unfortunately.

Yes, just feeding inputs 1 by 1 for each dequeue event until you get the number of outputs you want in your steady state is the best way. It minimizes memory usage. I'll see about updating the MDN documentation to state this better.

> Ideally I'd like to be able to set the CBR / VBR bitrate

What's wrong with the existing VBR/CBR modes? https://developer.mozilla.org/en-US/docs/Web/API/VideoEncode...

> I don't think you need a shader...

Ah I see what you mean. It'd probably be hard for us to standardize this in a way that worked across platforms which likely precludes us from doing anything quickly here. The stuff easiest to standardize for WebCodecs is stuff that's already standardized as part of the relevant codec spec (e.g, AVC, AV1, etc) and well supported on a significant range of hardware.

> ... instead of round-tripping into a CPU buffer

We're working on optimizing this in 2024, we do avoid CPU buffers in some cases, but not as many as we could.

* Hmm, what kind of scheme are you thinking beyond per frame QP? Does an abstraction on top of QP work for the case you have in mind?

* Reference frame control seems to be https://github.com/w3c/webcodecs/issues/285, there's some interest in this for 2024, so I'd expect progress here.

* Does splitting frames in WebGPU/WebGL work for the use case here? I'm not sure we could do anything internally (we're at the mercy of hardware decode implementations) without implementing such a shader.

Please file an issue at https://crbug.com/new with the details and we can take a look. Are you rendering frames in order?

Android may have some quirks due to legacy MediaCodec restrictions around how we more commonly need frames for video elements, frames only work in sequential order since they must be released to an output texture to access them (and releasing invalidates prior frames to speed up very old MediaCodecs).

The amount of frames necessary is going to depend on the codec and bitstream parameters. If it's H264 or H265, there's some more discussion and links here: https://github.com/w3c/webcodecs/issues/698#issuecomment-161...

The optimizeForLatency parameter may also help in some cases: https://developer.mozilla.org/en-US/docs/Web/API/VideoDecode...

Thanks for the nice write up! I work on the WebCodecs team at Chrome. I'm glad to hear it's mostly working for you. If you (or anyone else) has specific requests for new knobs regarding "We may need more encoding options, like non-reference frames or SVC", please file issues at https://github.com/w3c/webcodecs/issues

We're starting to see a move towards this with HEIF / AVIF containers, however in cases where "every bit must be saved" the general purpose containers like ISO-BMFF introduce some wastage that is unappealing.

It may just work in Firefox on OS with xHE-AAC support since they already rely on the system for AAC decoding IIRC. canPlayType/isTypeSupported probably indicate no support unless they've explicitly added support though.

I'm not aware of any hardware acceleration.

Chrome has supported xHE-AAC since 2020 on Android, 2022 on macOS, and added support on Windows last week.

It's a plug-in to the Windows ID3D11VideoProcessor API, https://source.chromium.org/chromium/chromium/src/+/main:ui/...

I think you can skip the transcode step entirely in favor of transmux with BMP too.

Chrome has something similar here: https://source.chromium.org/chromium/chromium/src/+/main:chr...

We accept contributions from other sites (e.g., Vimeo, Dailymotion, etc) who wanted to replace their long tail of in the wild embeds prior to Flash removal.

Yes, it was originally for decoding information only.

https://developer.mozilla.org/en-US/docs/Web/API/MediaCapabi... is on the way. I believe the latest Safari preview has it implemented and Chromium should have it shortly as well.

+1 to changing maybe to a shrug :)

MediaCapabilities is the new way to ask the "can I play this" question: https://developer.mozilla.org/en-US/docs/Web/API/Media_Capab...

It returns a boolean for whether a codec is supported or not.

FWIW, MediaCapabilities is the new way to ask the "can I play this" question:

https://developer.mozilla.org/en-US/docs/Web/API/Media_Capab...

It returns a boolean for whether a codec is supported or not.

Neat project! You could use WebCodec's ImageDecoder plus some custom controls to do this all in JavaScript without transcoding if you wanted. Otherwise you could use VideoEncoder and MSE for native video support.

https://developer.mozilla.org/en-US/docs/Web/API/WebCodecs_A... https://w3c.github.io/webcodecs/samples/ https://developer.mozilla.org/en-US/docs/Web/API/Media_Sourc...