Yeah. I was trying to do something like render a scene and encode it on-GPU before sending it to disk. Unfortunately, there’s no way to give FFMPEG video textures afaict so even if it uses hw decode, you’re copying the full pixel buffer data twice instead of the drastically smaller encoded frame once (or at least there’s no mechanism exposed that the JS ecosystem can make use of which relies on command-line ffmpeg)

There are OpenGL extensions which can import a provided GPU buffer as a texture, using those you can achieve zero-copy.

For instance, with VAAPI->OpenGL you would use vaExportSurfaceHandle in conjunction with glEGLImageTargetTexture2DOES.

Check out the "hwdec" mechanism in MPV:

https://github.com/mpv-player/mpv/blob/master/video/out/hwde...

https://github.com/mpv-player/mpv/blob/master/video/out/hwde...

Sure. I think the part that’s missing is that FFMPEG runs out of process and doesn’t deal in GPU buffers/textures.

If you use ffmpeg’s libavcodec interface then you can get it to give you the decoded framebuffers as exported DRM-prime descriptors which you can turn into textures. This is how Firefox does video decode with VAAPI, using libavcodec as a wrapper.

Edit: missed the part about JS ecosystem. You can move DRM prime descriptors between processes, but I assume you can’t do this from the ffmpeg CLI and would need to write your own little C wrapper around libavcodec

As someone else said you can achieve this with openGL, I also think gstreamer can do this.

I had a professional project I wanted to do in Linux but ended up going with Windows because of the 3D / video performance.

The API you have linked is too low level to my taste.

When I wanted to play video and render them inside D3D12 scenes, I have used a higher-level one, IMFMediaEngine https://learn.microsoft.com/en-us/windows/win32/api/mfmediae...

That OS-supplied high-level video player object can demux containers, decode, play audio, and copy uncompressed frames to D3D11 textures. Then, it’s relatively easy to use DXGI surface sharing API to share the resulting RGBA8 frames from D3D11 into D3D12.