This seems very strange to me. It’s how things would work with wl_shm, which is the baseline pixel-pushing interface in Wayland, but AFAIU Gtk uses EGL / Mesa, which in turn uses Linux dmabufs, which is how you do hardware-accelerated rendering / DRI on Linux today in general.
However, how precisely Linux dmabufs work in a DRI context is not clear to me, because the documentation is lacking, to say the least. It seems that you can ask map to dmabufs into memory, and you can create EGLSurfaces from them, but are they always mapped into CPU memory (if only kernel-side), or can they be bare GPU memory handles until the user asks to map them?
I’d hope for the latter, and if so, the only thing the work discussed in the article avoids is extra GPU-side blits (video decoding buffer to window buffer to screen), which is non-negligible but not necessarily the end of the world.
I believe GL_TEXTURE_EXTERNAL_OES is an Android-only OpenGL extension that takes the place of some uses of DMABUF but is not as flexible and general.
ChatGPT seems to know more about them, but I can't guarantee how accurate and up-to-date it is:
https://chat.openai.com/share/abff036b-3020-4093-a13b-86cbf0...
The tricky bit may be teaching pytorch to accept dmabuf handles and read and write dmabuf GPU buffers. (And ffmpeg too!)
Also, I wrote a significant part of GTK's current OpenGL renderer.
> But GPUs change the picture entirely. From what I understand by reading the article, GTK uses GL to render in the GPU then copies the pixels into main memory for the compositor to mix with other windows.
This is absolutely and completely incorrect. Once we get things into GL, the texture is backed by a DMABUF on Linux. You never read it back into main memory. That would be very, very, very stupid.
> But in modern GPU-first systems, the compositor is running in the GPU, so there would be no reason to ping-pong the pixels back and forth between CPU and GPU memory after drawing 3D or even 2D graphics with the GPU, even when having different processes draw and render the same pixels.
Yes, the compositor is running in the GPU too. So of course we just tell the compositor what the GL texture id is and it composites if it cannot map that texture (again, because it's really a DMABUF) as a toplevel plane for hardware scanout without using 3d capabilities at all.
That doesn't mean unaccelerrated. It means it doesn't power up the 3d part of the GPU. It's the fastest way in/out with the least power. You can avoid "compositing" from a compositor too when things are done right.
> So I'm afraid Wayland still has a lot of catching up to do, if it still uses a software compositor, and has to copy pixels back from the GPU that it drew with OpenGL. (Which is what I interpret the article as saying.)
Again, completely wrong.
> Check out iOS/macOS's "IOSurface":
Fun fact, I wrote the macos backend for GTK too. And yes, it uses IOSurface just like DMABUF works on Linux.
Could you please share your opinion on the toolkit, and its relation to others? Also, I heard that there were quite a lot of tech debt in GTK3 and part of the reason why GTK4 came as a bigger update is to fix those — what would you say, was it successful? Or is there still some legacy decisions that harm the project somewhat?
GTK 3 itself was trying to lose the tech debt of 2.x (which in turn 1.x). But they were still all wrapping a fundamentally crap API of X11 for graphics in this century.
GTK 4 changed that, and it now wraps a Wayland model of API. That drastically simplified GDK, which is why I could write a macOS backend in a couple of weeks.
It also completely changed how we draw. We no longer do immediate mode style (in the form of Cairo) and instead do a retained mode of draw commands. That allows for lots of new things you just couldn't do before with the old drawing model. It will also allow us to do a lot more fun things in the future (like threaded/tiled renderers).
The APIs all over the place were simplified and focused. I can't imagine writing an application the size of GNOME Builder again with anything less than GTK 4.
Hell, last GNOME cycle I rewrote Sysprof from scratch in a couple months, and it's become my co-pilot every day.
I'm sorry, I misinterpreted the paragraph in the article saying "exports" as meaning that it exports the pixels from GPU memory to CPU memory, not just passing a reference like GL_TEXTURE_EXTERNAL_OES and IOSurface does.
>GTK has already been using dmabufs since 4.0: When composing a frame, GTK translates all the render nodes (typically several for each widget) into GL commands, sends those to the GPU, and mesa then exports the resulting texture as a dmabuf and attaches it to our Wayland surface.
Perhaps I'd have been less confused if it said "passes a reference to the resulting texture in GPU memory" instead of "exports the resulting texture", because "exports" sounds expensive to me.
Out of curiosity about the big picture, are dmabufs a Linux thing that's independent of OpenGL, or independent of the device driver, or build on top of GL_TEXTURE_EXTERNAL_OES, or is GL_TEXTURE_EXTERNAL_OES/SurfaceTexture just an Android or OpenGL ES thing that's an alternative to dmabufs in Linux? Do they work without any dependencies on X or Wayland or OpenGL, I hope? (Since pytorch doesn't use OpenGL.)
https://source.android.com/docs/core/graphics/arch-st
One practical non-gui use case I have for passing references to GPU textures between processes on Linux is pytorch: I'd like to be able to decompress video in one process or docker container on a cloud instance with an NVidia accelerator, and then pass zero-copy references to the resulting frames into another process (or even two -- each frame of video needs to be run through two different vision models) in another docker container running pytorch, sharing and multitasking the same GPU, possibly sending handles through a shared local file system or ipc (like how IOSurface uses Mach messages to magically send handles, or using unix domain sockets or ZeroMQ or something like that), but I don't know if it's something that's supported at the Linux operating system level (ubuntu), or if I'd have to drop down to the NVidia driver level to do that.
NVidia has some nice GPU video decompressor libraries, but they don't necessarily play well with pytorch in the same process, so I'd like to run them (or possibly ffmpeg) in a different process, but on the same GPU. Is it even possible, or am I barking up the wrong tree?
It would be ideal if ffmpeg had a built-in "headless" way to perform accelerated video decompression and push out GPU texture handles to other processes somehow, instead of rendering itself or writing pixels to files or touching CPU memory.
They are independent of the graphics subsystem altogether (although that is where they got their start, afaik). Your webcam also uses DMABUF. So if you want to display your webcam from a GTK 4 application, this GtkGraphicsOffload will help you take that DMABUF from your camera (which may not be mappable on CPU memory, but can DMA pass to your GPU), and display it in a GTK application. It could either be composited on the GPU, or mapped directly to scanout if the right conditions are met.
I wrote a library recently (libmks) and found the culprits in Qemu/VirGL/virtio_gpu that were preventing passing a DMABUF from inside a guest VM to the host. That stuff is all fixed now so theoretically you could even have a webcam in a VM which then uses a GTK 4 application to render with VirGL and the compositor submit the scene to the host OS which itself can set the planes correctly to get the same performance as if it were in the host OS.
> I'd like to be able to decompress video in one process or docker container on a cloud instance with an NVidia accelerator, and then pass zero-copy references
If you want this stuff with NVidia, and you're a customer, I highly suggest you tell your NVidia representative this. Getting them to use DMABUF in a fashion that can be used from other sub-systems would be fantastic.
But at it's core, if you were using Mesa and open drivers for some particular piece of hardware, yes it's capable of working given the right conditions.
https://en.wikipedia.org/wiki/SunView
Programmers Reference Manual for the Sun Window System, rev C of 1 November 1983: Page 23, Locking and Clipping:
http://bitsavers.trailing-edge.com/pdf/sun/sunos/1.0/800-109...
But GPUs change the picture entirely. From what I understand by reading the article, GTK uses GL to render in the GPU then copies the pixels into main memory for the compositor to mix with other windows. But in modern GPU-first systems, the compositor is running in the GPU, so there would be no reason to ping-pong the pixels back and forth between CPU and GPU memory after drawing 3D or even 2D graphics with the GPU, even when having different processes draw and render the same pixels.
So I'm afraid Wayland still has a lot of catching up to do, if it still uses a software compositor, and has to copy pixels back from the GPU that it drew with OpenGL. (Which is what I interpret the article as saying.)
More recently (on an archeological time scale, but for many years by now), MacOS, Windows, iOS, and Android have all developed ways of sharing graphics between multiple processes not only in shared main CPU memory, but also on the GPU, which greatly accelerates rendering, and is commonly used by web browsers, real time video playing and processing tools, desktop window managers, and user interface toolkits.
There are various APIs to pass handles to "External" or "IO Surface" shared GPU texture memory around between multiple processes. I've written about those APIs on Hacker News frequently over the years:
https://news.ycombinator.com/item?id=13534298
DonHopkins on Jan 31, 2017 | parent | context | favorite | on: Open-sourcing Chrome on iOS
It's my understanding that only embedded WKWebViews are allowed to enable the JIT compiler, but not UIWebViews (or in-process JavaScriptCore engines). WKWebView is an out-of-process web browser that uses IOSurface [1] to project the image into your embedding application and IPC to send messages.
So WKWebView's dynamically generated code is running safely firewalled in a separate address space controlled by Apple and not accessible to your app, while older UIWebViews run in the address space of your application, and aren't allowed to write to code pages, so their JIT compiler is disabled.
Since it's running in another process, WkWebView's JavaScriptEngine lacks the ability to expose your own Objective C classes to JavaScript so they can be called directly [2], but it does include a less efficient way of adding script message handlers that call back to Objective C code via IPC [3].
[1] https://developer.apple.com/reference/iosurface
[2] https://developer.apple.com/reference/javascriptcore/jsexpor...
[3] https://developer.apple.com/reference/webkit/wkusercontentco...
https://news.ycombinator.com/item?id=18763463
DonHopkins on Dec 26, 2018 | parent | context | favorite | on: WKWebView, an Electron alternative on macOS/iOS
Yes, it's a mixed bag with some things better and others worse. But having a great JavaScript engine with the JIT enabled is pretty important for many applications. But breaking up the browser into different processes and communicating via messages and sharing textures in GPU memory between processes (IOSurface, GL_TEXTURE_EXTERNAL_OES, etc) is the inextricable direction of progress, what all the browsers are doing now, and why for example Firefox had to make so many old single-process XP-COM xulrunner plug-ins obsolete.
IOSurface:
https://developer.apple.com/documentation/iosurface?language...
https://shapeof.com/archives/2017/12/moving_to_metal_episode...
GL_TEXTURE_EXTERNAL_OES:
https://developer.android.com/reference/android/graphics/Sur...
http://www.felixjones.co.uk/neo%20website/Android_View/
pcwalton on Dec 27, 2018 | prev [–]
Chrome and Firefox with WebRender are going the opposite direction and just putting all their rendering in the chrome process/"GPU process" to begin with.
DonHopkins on Dec 27, 2018 | parent [–]
Yes I know, that's exactly what I meant by "breaking up the browser into different processes". They used to all be in the same process. Now they're in different processes, and communicate via messages and shared GPU memory using platform specific APIs like IOSurface. So it's no longer possible to write an XP/COM plugin for the browser in C++, and call it from the renderer, because it's running in a different process, so you have to send messages and use shared memory instead. But then if the renderer crashes, the entire browser doesn't crash.
https://news.ycombinator.com/item?id=20313751
DonHopkins on June 29, 2019 | parent | context | favorite | on: Red Hat Expecting X.org to “Go into Hard Maintenan...
Actually, Electron (and most other web browsers) on the Mac OS/X and iOS use IOSurface to share zero-copy textures in GPU memory between the render and browser processes. Android and Windows (I presume, but don't know name of the API, probably part of DirectX) have similar techniques. It's like shared memory, but for texture memory in the GPU between separate heavy weight processes. Since simply sharing main memory between processes wouldn't be nearly as efficient, requiring frequent uploading and downloading textures to and from the GPU.
Mac OS/X and iOS IOSurface:
https://developer.apple.com/documentation/iosurface?language...
http://neugierig.org/software/chromium/notes/2010/08/mac-acc...
https://github.com/SimHacker/UnityJS/blob/master/notes/IOSur...
Android SurfaceTexture and GL_TEXTURE_EXTERNAL_OES:
https://developer.android.com/reference/android/graphics/Sur...
https://www.khronos.org/registry/OpenGL/extensions/OES/OES_E...
https://docs.google.com/document/d/1J0fkaGS9Gseczw3wJNXvo_r-...
https://github.com/SimHacker/UnityJS/blob/master/notes/ZeroC...
https://github.com/SimHacker/UnityJS/blob/master/notes/Surfa...
https://news.ycombinator.com/item?id=25997356
DonHopkins on Feb 2, 2021 | parent | context | favorite | on: VideoLAN is 20 years old today
>Probably do a multi-process media player, like Chrome is doing, with parsers and demuxers in a different process, and different ones for decoders and renderers. Knowing that you probably need to IPC several Gb/s between them. Chrome and other browsers and apps, and drivers like virtual webcams, and libraries like Syphon, can all pass "zero-copy" image buffers around between different processes by sharing buffers in GPU memory (or main memory too of course) and sending IPC messages pointing to the shared buffers.
That's how the browser's web renderer processes efficiently share the rendered images with the web browser user interface process, for example. And how virtual webcam drivers can work so efficiently, too.
Check out iOS/macOS's "IOSurface":
https://developer.apple.com/documentation/iosurface
>IOSurface Share hardware-accelerated buffer data (framebuffers and textures) across multiple processes. Manage image memory more efficiently.
>Overview: The IOSurface framework provides a framebuffer object suitable for sharing across process boundaries. It is commonly used to allow applications to move complex image decompression and draw logic into a separate process to enhance security.
And Android's "SurfaceTexture" and GL_TEXTURE_EXTERNAL_OES:
https://developer.android.com/reference/android/graphics/Sur...
>The image stream may come from either camera preview or video decode. A Surface created from a SurfaceTexture can be used as an output destination for the android.hardware.camera2, MediaCodec, MediaPlayer, and Allocation APIs. When updateTexImage() is called, the contents of the texture object specified when the SurfaceTexture was created are updated to contain the most recent image from the image stream. This may cause some frames of the stream to be skipped.
https://source.android.com/devices/graphics/arch-st
>The main benefit of external textures is their ability to render directly from BufferQueue data. SurfaceTexture instances set the consumer usage flags to GRALLOC_USAGE_HW_TEXTURE when it creates BufferQueue instances for external textures to ensure that the data in the buffer is recognizable by GLES.
And Syphon, which has a rich ecosystem of apps and tools and libraries:
http://syphon.v002.info
>Syphon is an open source Mac OS X technology that allows applications to share frames - full frame rate video or stills - with one another in realtime. Now you can leverage the expressive power of a plethora of tools to mix, mash, edit, sample, texture-map, synthesize, and present your imagery using the best tool for each part of the job. Syphon gives you flexibility to break out of single-app solutions and mix creative applications to suit your needs.
Of course there's a VLC Syphon server:
https://github.com/rsodre/VLCSyphon