News for hackers. Stuff that matters.
Re:GTK: Introducing Graphics Offload by smallstepforman on Saturday November 18, 2023 @07:00AM (#38318513)
BeOS and Haiku allowed exposure to kernel graphics buffers decades ago (https://www.haiku-os.org/legacy-docs/bebook/BDirectWindow.ht...), which bypass the roundtrip to compositor and back. A companion article describing the design is here (https://www.haiku-os.org/legacy-docs/benewsletter/Issue3-12....)

After 25 years, GTK joins the party …

With no native video drivers, moving a Vesa Haiku window with video playback still seems smoother than doing the same in Gnome/Kde under X11.

Re:GTK: Introducing Graphics Offload by pjmlp on Saturday November 18, 2023 @07:17AM (#38318672)
Like most OSes where the desktop APIs are part of the whole developer experience, and not yet another pluggable piece.
Re:GTK: Introducing Graphics Offload by tmountain on Saturday November 18, 2023 @07:18AM (#38318682)
BeOS was truly ahead of its time. It’s a shame that it didn’t get more traction.
Re:GTK: Introducing Graphics Offload by ris on Saturday November 18, 2023 @10:19AM (#38320508)
There must be a name for the logical fallacy I see whenever someone pines over a past "clearly superior" technology that wasn't adopted or had its project cancelled, but I can never grasp it. I guess the closest thing to it is an unfalsifiable statement.

The problem comes from people remembering all of the positive traits (often just promises) of a technology but either forgetting all the problems it had or the technology never being given enough of a chance for people to discover all the areas in which it is a bit crap.

BeOS? Impressive technology demo. Missing loads of features expected in a modern OS, even at that time. No multi-user model even!

This is also a big phenomenon in e.g. aviation. The greatest fighter jet ever is always the one that (tragically!) got cancelled before anyone could discover its weaknesses.

Re:GTK: Introducing Graphics Offload by AshamedCaptain on Saturday November 18, 2023 @10:39AM (#38320703)
> There must be a name for the logical fallacy I see whenever someone pines over a past "clearly superior" technology that wasn't adopted or had its project cancelled, but I can never think of it. I guess the closest thing to it is an unfalsifiable statement.

You can run Haiku today, so it's hardly unfalsifiable, nor an effect of nostalgia or whatever way you want to phrase it.

> BeOS? Impressive technology demo. Missing loads of features expected in a modern OS, even at that time. No multi-user model even!

"Even at that time" is just false. multi-user safe OSes abound in the 90s?

Re:GTK: Introducing Graphics Offload by dihrbtk on Saturday November 18, 2023 @11:52AM (#38321513)
Windows NT...?
Re:GTK: Introducing Graphics Offload by ris on Saturday November 18, 2023 @11:53AM (#38321526)
> You can run Haiku today, so it's hardly unfalsifiable

Excellent, so let's falsify it: how come 20 years later the best thing people really have to say about BeOS/Haiku is that is has smooth window dragging?

> multi-user safe OSes abound in the 90s?

Windows NT.

Linux and at least two other free unixes.

Countless proprietary unixes.

VMS.

The widespread desktop OSs in the 90s were not considered serious OSs even then, more accidents of backward-compatibility needs.

Re:GTK: Introducing Graphics Offload by tialaramex on Saturday November 18, 2023 @03:11PM (#38323838)
The thing about Haiku is that their plan (initially as "OpenBeOS") was since they're just re-doing BeOS and some of BeOS is Open Source they'll just get some basics working then they're right back on the horse and in a few years they'll be far ahead of where BeOS was.

Over two decades later they don't have a 1.0 release.

Re:GTK: Introducing Graphics Offload by darkwater on Saturday November 18, 2023 @10:38AM (#38320697)
This, plus nostalgia pink glasses plus rooting for the underdog (who lost) plus my niche tech is better than your mainstream one.
Re:GTK: Introducing Graphics Offload by cmrdporcupine on Saturday November 18, 2023 @11:54AM (#38321537)
Exactly this. I'm as big a fan of "alternative tech timelines" as the next nerd, but I also can see in retrospect why we have the set of compromises we have today, and all along I watched the intense efforts people made to navigate the mindbogglingly complicated minefield of competing approaches and political players and technical innovations that were on the scene.

People have been working damned hard to build things like Gtk, Qt, etc. not to mention Wayland, etc. all the while maintaining compatibility etc and I personally am happy for their efforts.

BeOS/HaikuOS is a product of a mid-90s engineering scene that predates the proliferation of GPUs, the web, and the set of programming languages that we work with today. There's nothing wrong with it in that context, but it's also not "better." Just different compromises.

The other one I see nostalgia nerds reach for is the Amiga. A system highly coupled to a set of custom chips that only made sense when RAM was as fast (or faster) than the CPU, whose OS had no memory protection, and which was easily outstripped in technical abilities by the early 90s by PCs with commodity ISA cards, etc. because of the development of economics of scale in computer manufacturing, etc. It was ahead of its time for about 2-3 years in the mid 80s, but in a way that was a dead end.

Anyways, what we have right now is a messy set of compromises. It doesn't hurt to go looking for simplifications, but it does hurt to pretend that the compromises don't exist for a reason.

EDIT: I would add though that "multiuser" as part of a desktop (or especially mobile) OS has maybe proven to be a pointless thing. The vast majority of Linux machines out there in the world are run in a single user fashion, even if they are capable of multiuser. Android phones, Chromebooks, desktop machines, and even many servers -- mostly run with just one master user. And we've also seen how rather not-good the Unix account permissions model is in terms of security, and how weak its timesharing of resources etc is in context of today's needs -- hence the development of cgroups, containers, and virtual machine / hypervisor etc.

Re:GTK: Introducing Graphics Offload by BenjiWiebe on Saturday November 18, 2023 @01:08PM (#38322448)
They run with one master user perhaps, but they have multiple users at one time anyways.
Re:GTK: Introducing Graphics Offload by cmrdporcupine on Saturday November 18, 2023 @01:29PM (#38322760)
I mean, in those cases they're almost always just using multiple users as a proxy for job authorization levels, not people.

Anybody who is serious about securing a single physical machine for multiuser access isn't doing it through multiple OS accounts, and is slicing it up by VMs, instead.

I do have a home (Windows) computer that gets used by multiple family members through accounts, but I think this isn't a common real world use case.

Re:GTK: Introducing Graphics Offload by pjmlp on Saturday November 18, 2023 @12:00PM (#38321631)
Unfortunately being technology superior isn't enough to win.

Money, marketing, politics, wanting to be part of the crowd, usually play a bigger role.

Re:GTK: Introducing Graphics Offload by timetraveller26 on Saturday November 18, 2023 @01:27PM (#38322740)
I was greatly impressed with BeOS filesystem SQL'esque indexing and querying.
Re:GTK: Introducing Graphics Offload by fulafel on Saturday November 18, 2023 @07:49AM (#38318953)
You can bypass the compositor with X11 hw acceleration features too. But what about Wayland? I thought apps always go through the compositor there. As shown eg in the diagram at https://www.apertis.org/architecture/wayland_compositors/

Drawing directly from app to kernel graphics buffers (or hw backed surfaces) and participating in composition are orthogonal I think. The compositor may be compositing the kernel or hw backed surfaces.

Re:GTK: Introducing Graphics Offload by arghwhat on Saturday November 18, 2023 @08:25AM (#38319294)
Bypassing composition is a key feature in Wayland. The protocols are all written with zero-copy in mind. The X11 tricks are already there, forwarding one clients buffers straight to hardware, while the end-game with libliftoff is offloading multiple subsurfaces directly to individual planes at once.

The compositor is the entire display server in Wayland, and is also the component responsible for bypassing composition when possible.

Re:GTK: Introducing Graphics Offload by fulafel on Saturday November 18, 2023 @12:35PM (#38322055)
In my meager understanding which I'm happy to be corrected about: In a windowed scenario (vs fullscreen), in both the X direct-rendering and Wayland scenarios the application provides a (possibly gpu backed) surface that the compositor uses as a texture when forming the full screen video output.

In a full-screen case AFAIK it's possible to skip the compositing step with X11, and maybe with Wayland too.

"Zero copy" seems a bit ambiguous term in graphics because there's the kind of copying where whole screen or window sized buffers are being copied about, and then there are compositing operations where various kinds of surfaces are inputs to the final rendering, where also pixels are copied around possibly several times in shaders but there aren't necessary extra texture sized intermediate buffers involved.

Re:GTK: Introducing Graphics Offload by arghwhat on Saturday November 18, 2023 @03:32PM (#38324062)
> In a full-screen case AFAIK it's possible to skip the compositing step with X11, and maybe with Wayland too.

This is the trivial optimization all Wayland compositors do.

The neater trick is to do this for non-fullscreen content - and even just parts of windows - using overlay planes. Some compositors have their own logic for this, but libliftoff aims to generalize it.

Zero-copy is not really ambiguous, but to clarify: Wayland protocols are designed to maximize the cases where a buffer rendered by a client can be presented directly by the display hardware as-is (scanned out), without any intermediate operations on the content.

Note "maximize" - the content must be compatible with hardware capabilities. Wayland provides hints to stay within capabilities, but a client might pick a render buffer modifier/format that cannot be scanned out by the display hardware. GPUs have a lot of random limitations.

Re:GTK: Introducing Graphics Offload by AshamedCaptain on Saturday November 18, 2023 @08:16AM (#38319188)
To this day, moving a Haiku window under a 5k unaccelerated EFI GOP framebuffer still feels _significantly_ faster than doing the same under Windows 11, and everything KWin's X/Wayland has to offer on the same hardware (AMD Navi 2 GPU).

> BeOS and Haiku allowed exposure to kernel graphics buffers decades ago

In any case, X also allowed this for ages, with XV and XShm and the like. Of course then everyone got rid of this in order to have the GPU in the middle for fancier animations and whatnot, and things went downhill since.

[dead]
Re:GTK: Introducing Graphics Offload by ElectricalUnion on Saturday November 18, 2023 @06:41PM (#38326188)
Hardware acceleration in this case is to draw less power doing day-to-day operations and not melt the user's hand.

In the general case most effective hardware acceleration uses are supposed to increase latency.

Let's take another example for the analogy: you can run your LLM straight in the CPU and it will have awesome latency and very poor throughput; or you can take your sweet time to load the weights into your GPU/NPU/TPU/memory-in-compute device and run it hardware accelerated at lower energy cost and higher throughput.

Re:GTK: Introducing Graphics Offload by play_ac on Saturday November 18, 2023 @09:14AM (#38319820)
>In any case, X also allowed this for ages, with XV and XShm and the like.

No, XShm doesn't do that and the way XV does it is completely dependent on drivers. If you're using Glamour then XV won't use overlays at all. XShm uses a buffer created in CPU memory allocated by the client that the X server then has to copy to the screen.

> Of course then everyone got rid of this in order to have the GPU in the middle for fancier animations

No, for video, the GPU is used in the middle so you can do post-processing without copying everything into main memory and stalling the whole pipeline. I'd like to see an actual benchmark for how a fullscreen 5k video with post-processing plays on Haiku without any hardware acceleration.

Re:GTK: Introducing Graphics Offload by AshamedCaptain on Saturday November 18, 2023 @09:34AM (#38320053)
> XShm uses a buffer created in CPU memory allocated by the client that the X server then has to copy to the screen.

This is completely wrong. The _entire point_ of XShm, as the name implies, is to have _shared memory_ between the client and the X server. And the server controls the card's framebuffer (at least, in traditional X). There's absolutely no copy and that was the entire point of the extension.

(if compositing is enabled, and therefore the GPU is in between, the server eventually has to call texture_from_surface; at that point, whether there is a copy or not depends on the GPU's driver).

You could literally read the PREVIOUS contents of your card's framebuffer by doing XShmGetImage on the root window.

> No, for video, the GPU is used in the middle so you can do post-processing without copying everything into main memory and stalling the whole pipeline.

You can do many types of card-accelerated zero-copy post-processing using XV: colorspace conversion, scaling, etc., which, again, was the point of the extension. At the time, scaling the video in software or even just doing a memory copy per frame would have completely tanked frame rate. How do you think I used Xine to watch DVDs on Pentium II-level hardware? Obviously you cannot put the video in faces of rotating 3D cube, but this is precisely what I call "fancy animations".

Re:GTK: Introducing Graphics Offload by AshamedCaptain on Saturday November 18, 2023 @10:13AM (#38320452)
> XShm uses a buffer created in CPU memory allocated by the client that the X server then has to copy to the screen.

Fair enough. Even with XShmCreatePixmap, you are still never simply mmaping the card's actual entire framebuffer, unlike what BDirectWindow allows (if https://www.haiku-os.org/legacy-docs/benewsletter/Issue3-12.... is to believed, which is closer to something like DGA). In XShm, the server still has to copy your shared memory segment to the actual framebuffer.

(sorry for previous answer here, I misunderstood your comment)

> No, for video, the GPU is used in the middle so you can do post-processing without copying everything into main memory and stalling the whole pipeline.

Depends on what you mean by "post-processing". You can do many types of card-accelerated zero-copy post-processing using XV: colorspace conversion, scaling, etc. At the time, scaling the video in software or even just doing an extra memory copy per frame would have tanked frame rate -- Xine can be used to watch DVDs in PentiumII-level hardware. Obviously you cannot put the video in the faces of rotating 3D cubes, but this is precisely what I call "fancy animations".

[dead]
Re:GTK: Introducing Graphics Offload by AshamedCaptain on Saturday November 18, 2023 @11:16AM (#38321112)
I don't see what you mean by "condescending remarks", but I do think it is stretching it to claim "arbitrary filters" is a "basic feature that a video player is required to support now". As a consumer, I have absolutely _never_ used any such video filters, doubt most consumers are even aware of them, have seen few video players which support them, and most definitely I have no idea what they are in VLC. Do they even enable any video filters by default? The only video filter I have sometimes used is deinterlacing which doesn't really fit well in the GPU rendering pipeline anyway but fits very nicely in fixed hardware. So yes, I hardly see the need to stop using native accelerated video output and fallback to GPU just in case someone wants to use such filters. This is how I end up with a card which consumes 20W just for showing a static desktop on two monitors.

Anyway, discussing about this is besides the point, and forgive me from the rant above.

If you really need GPU video filters then the GPU is obviously going to be the best way to implement them, there's no discussion possible about that. But the entire point of TFA is to (dynamically) go back to a model where the GPU is _not_ in the middle. And that model -- sans GPU -- happens to match what Xv was doing and is actually faster and less power consuming than to always blindly use the GPU which is where we are now post-Xv.

Re:GTK: Introducing Graphics Offload by DonHopkins on Saturday November 18, 2023 @08:57AM (#38319642)
That's also how SunView worked on SunOS in 1982, and Sun's later GX graphics accelerated framebuffer driver worked in the 90's. The kernel managed the clipping list, and multiple processes shared the same memory, locking and respecting the clipping list and pixel buffers in shared memory (main memory, not GPU memory!), so multiple process could draw on different parts of the screen efficiently, without incurring system calls and context switches.

https://en.wikipedia.org/wiki/SunView

Programmers Reference Manual for the Sun Window System, rev C of 1 November 1983: Page 23, Locking and Clipping:

http://bitsavers.trailing-edge.com/pdf/sun/sunos/1.0/800-109...

But GPUs change the picture entirely. From what I understand by reading the article, GTK uses GL to render in the GPU then copies the pixels into main memory for the compositor to mix with other windows. But in modern GPU-first systems, the compositor is running in the GPU, so there would be no reason to ping-pong the pixels back and forth between CPU and GPU memory after drawing 3D or even 2D graphics with the GPU, even when having different processes draw and render the same pixels.

So I'm afraid Wayland still has a lot of catching up to do, if it still uses a software compositor, and has to copy pixels back from the GPU that it drew with OpenGL. (Which is what I interpret the article as saying.)

More recently (on an archeological time scale, but for many years by now), MacOS, Windows, iOS, and Android have all developed ways of sharing graphics between multiple processes not only in shared main CPU memory, but also on the GPU, which greatly accelerates rendering, and is commonly used by web browsers, real time video playing and processing tools, desktop window managers, and user interface toolkits.

There are various APIs to pass handles to "External" or "IO Surface" shared GPU texture memory around between multiple processes. I've written about those APIs on Hacker News frequently over the years:

https://news.ycombinator.com/item?id=13534298

DonHopkins on Jan 31, 2017 | parent | context | favorite | on: Open-sourcing Chrome on iOS

It's my understanding that only embedded WKWebViews are allowed to enable the JIT compiler, but not UIWebViews (or in-process JavaScriptCore engines). WKWebView is an out-of-process web browser that uses IOSurface [1] to project the image into your embedding application and IPC to send messages.

So WKWebView's dynamically generated code is running safely firewalled in a separate address space controlled by Apple and not accessible to your app, while older UIWebViews run in the address space of your application, and aren't allowed to write to code pages, so their JIT compiler is disabled.

Since it's running in another process, WkWebView's JavaScriptEngine lacks the ability to expose your own Objective C classes to JavaScript so they can be called directly [2], but it does include a less efficient way of adding script message handlers that call back to Objective C code via IPC [3].

[1] https://developer.apple.com/reference/iosurface

[2] https://developer.apple.com/reference/javascriptcore/jsexpor...

[3] https://developer.apple.com/reference/webkit/wkusercontentco...

https://news.ycombinator.com/item?id=18763463

DonHopkins on Dec 26, 2018 | parent | context | favorite | on: WKWebView, an Electron alternative on macOS/iOS

Yes, it's a mixed bag with some things better and others worse. But having a great JavaScript engine with the JIT enabled is pretty important for many applications. But breaking up the browser into different processes and communicating via messages and sharing textures in GPU memory between processes (IOSurface, GL_TEXTURE_EXTERNAL_OES, etc) is the inextricable direction of progress, what all the browsers are doing now, and why for example Firefox had to make so many old single-process XP-COM xulrunner plug-ins obsolete.

IOSurface:

https://developer.apple.com/documentation/iosurface?language...

https://shapeof.com/archives/2017/12/moving_to_metal_episode...

GL_TEXTURE_EXTERNAL_OES:

https://developer.android.com/reference/android/graphics/Sur...

http://www.felixjones.co.uk/neo%20website/Android_View/

pcwalton on Dec 27, 2018 | prev [–]

Chrome and Firefox with WebRender are going the opposite direction and just putting all their rendering in the chrome process/"GPU process" to begin with.

DonHopkins on Dec 27, 2018 | parent [–]

Yes I know, that's exactly what I meant by "breaking up the browser into different processes". They used to all be in the same process. Now they're in different processes, and communicate via messages and shared GPU memory using platform specific APIs like IOSurface. So it's no longer possible to write an XP/COM plugin for the browser in C++, and call it from the renderer, because it's running in a different process, so you have to send messages and use shared memory instead. But then if the renderer crashes, the entire browser doesn't crash.

https://news.ycombinator.com/item?id=20313751

DonHopkins on June 29, 2019 | parent | context | favorite | on: Red Hat Expecting X.org to “Go into Hard Maintenan...

Actually, Electron (and most other web browsers) on the Mac OS/X and iOS use IOSurface to share zero-copy textures in GPU memory between the render and browser processes. Android and Windows (I presume, but don't know name of the API, probably part of DirectX) have similar techniques. It's like shared memory, but for texture memory in the GPU between separate heavy weight processes. Since simply sharing main memory between processes wouldn't be nearly as efficient, requiring frequent uploading and downloading textures to and from the GPU.

Mac OS/X and iOS IOSurface:

https://developer.apple.com/documentation/iosurface?language...

http://neugierig.org/software/chromium/notes/2010/08/mac-acc...

https://github.com/SimHacker/UnityJS/blob/master/notes/IOSur...

Android SurfaceTexture and GL_TEXTURE_EXTERNAL_OES:

https://developer.android.com/reference/android/graphics/Sur...

https://www.khronos.org/registry/OpenGL/extensions/OES/OES_E...

https://docs.google.com/document/d/1J0fkaGS9Gseczw3wJNXvo_r-...

https://github.com/SimHacker/UnityJS/blob/master/notes/ZeroC...

https://github.com/SimHacker/UnityJS/blob/master/notes/Surfa...

https://news.ycombinator.com/item?id=25997356

DonHopkins on Feb 2, 2021 | parent | context | favorite | on: VideoLAN is 20 years old today

>Probably do a multi-process media player, like Chrome is doing, with parsers and demuxers in a different process, and different ones for decoders and renderers. Knowing that you probably need to IPC several Gb/s between them. Chrome and other browsers and apps, and drivers like virtual webcams, and libraries like Syphon, can all pass "zero-copy" image buffers around between different processes by sharing buffers in GPU memory (or main memory too of course) and sending IPC messages pointing to the shared buffers.

That's how the browser's web renderer processes efficiently share the rendered images with the web browser user interface process, for example. And how virtual webcam drivers can work so efficiently, too.

Check out iOS/macOS's "IOSurface":

https://developer.apple.com/documentation/iosurface

>IOSurface Share hardware-accelerated buffer data (framebuffers and textures) across multiple processes. Manage image memory more efficiently.

>Overview: The IOSurface framework provides a framebuffer object suitable for sharing across process boundaries. It is commonly used to allow applications to move complex image decompression and draw logic into a separate process to enhance security.

And Android's "SurfaceTexture" and GL_TEXTURE_EXTERNAL_OES:

https://developer.android.com/reference/android/graphics/Sur...

>The image stream may come from either camera preview or video decode. A Surface created from a SurfaceTexture can be used as an output destination for the android.hardware.camera2, MediaCodec, MediaPlayer, and Allocation APIs. When updateTexImage() is called, the contents of the texture object specified when the SurfaceTexture was created are updated to contain the most recent image from the image stream. This may cause some frames of the stream to be skipped.

https://source.android.com/devices/graphics/arch-st

>The main benefit of external textures is their ability to render directly from BufferQueue data. SurfaceTexture instances set the consumer usage flags to GRALLOC_USAGE_HW_TEXTURE when it creates BufferQueue instances for external textures to ensure that the data in the buffer is recognizable by GLES.

And Syphon, which has a rich ecosystem of apps and tools and libraries:

http://syphon.v002.info

>Syphon is an open source Mac OS X technology that allows applications to share frames - full frame rate video or stills - with one another in realtime. Now you can leverage the expressive power of a plethora of tools to mix, mash, edit, sample, texture-map, synthesize, and present your imagery using the best tool for each part of the job. Syphon gives you flexibility to break out of single-app solutions and mix creative applications to suit your needs.

Of course there's a VLC Syphon server:

https://github.com/rsodre/VLCSyphon

Re:GTK: Introducing Graphics Offload by mananaysiempre on Saturday November 18, 2023 @10:22AM (#38320538)
> From what I understand by reading the article, GTK uses GL to render in the GPU then copies the pixels into main memory for the compositor to mix with other windows.

This seems very strange to me. It’s how things would work with wl_shm, which is the baseline pixel-pushing interface in Wayland, but AFAIU Gtk uses EGL / Mesa, which in turn uses Linux dmabufs, which is how you do hardware-accelerated rendering / DRI on Linux today in general.

However, how precisely Linux dmabufs work in a DRI context is not clear to me, because the documentation is lacking, to say the least. It seems that you can ask map to dmabufs into memory, and you can create EGLSurfaces from them, but are they always mapped into CPU memory (if only kernel-side), or can they be bare GPU memory handles until the user asks to map them?

I’d hope for the latter, and if so, the only thing the work discussed in the article avoids is extra GPU-side blits (video decoding buffer to window buffer to screen), which is non-negligible but not necessarily the end of the world.

Re:GTK: Introducing Graphics Offload by rjsw on Saturday November 18, 2023 @10:45AM (#38320768)
Linux on ARM SoCs with HW video decoders that are separate to the GPU can use the V4L2 API to avoid some copying. The decoder writes a frame to a buffer that the GPU can see then you use GL to get the GPU to merge it into the framebuffer.
Re:GTK: Introducing Graphics Offload by DonHopkins on Saturday November 18, 2023 @04:15PM (#38324532)
I misunderstood the article saying "exports the resulting texture" as meaning it exported it to CPU memory, but audidude explained how it actually works.

I believe GL_TEXTURE_EXTERNAL_OES is an Android-only OpenGL extension that takes the place of some uses of DMABUF but is not as flexible and general.

ChatGPT seems to know more about them, but I can't guarantee how accurate and up-to-date it is:

https://chat.openai.com/share/abff036b-3020-4093-a13b-86cbf0...

The tricky bit may be teaching pytorch to accept dmabuf handles and read and write dmabuf GPU buffers. (And ffmpeg too!)

Re:GTK: Introducing Graphics Offload by audidude on Saturday November 18, 2023 @10:34AM (#38320655)
I can't respond to everything incorrect in this, because it's way to long to read. But from the very start...

Also, I wrote a significant part of GTK's current OpenGL renderer.

> But GPUs change the picture entirely. From what I understand by reading the article, GTK uses GL to render in the GPU then copies the pixels into main memory for the compositor to mix with other windows.

This is absolutely and completely incorrect. Once we get things into GL, the texture is backed by a DMABUF on Linux. You never read it back into main memory. That would be very, very, very stupid.

> But in modern GPU-first systems, the compositor is running in the GPU, so there would be no reason to ping-pong the pixels back and forth between CPU and GPU memory after drawing 3D or even 2D graphics with the GPU, even when having different processes draw and render the same pixels.

Yes, the compositor is running in the GPU too. So of course we just tell the compositor what the GL texture id is and it composites if it cannot map that texture (again, because it's really a DMABUF) as a toplevel plane for hardware scanout without using 3d capabilities at all.

That doesn't mean unaccelerrated. It means it doesn't power up the 3d part of the GPU. It's the fastest way in/out with the least power. You can avoid "compositing" from a compositor too when things are done right.

> So I'm afraid Wayland still has a lot of catching up to do, if it still uses a software compositor, and has to copy pixels back from the GPU that it drew with OpenGL. (Which is what I interpret the article as saying.)

Again, completely wrong.

> Check out iOS/macOS's "IOSurface":

Fun fact, I wrote the macos backend for GTK too. And yes, it uses IOSurface just like DMABUF works on Linux.

Re:GTK: Introducing Graphics Offload by kaba0 on Saturday November 18, 2023 @12:14PM (#38321808)
I’m not the parent poster, I’m just tryin go to grab this opportunity that I “met” someone so familiar with GTK :)

Could you please share your opinion on the toolkit, and its relation to others? Also, I heard that there were quite a lot of tech debt in GTK3 and part of the reason why GTK4 came as a bigger update is to fix those — what would you say, was it successful? Or is there still some legacy decisions that harm the project somewhat?

Re:GTK: Introducing Graphics Offload by audidude on Saturday November 18, 2023 @01:34PM (#38322828)
> Also, I heard that there were quite a lot of tech debt in GTK3 and part of the reason why GTK4 came as a bigger update is to fix those

GTK 3 itself was trying to lose the tech debt of 2.x (which in turn 1.x). But they were still all wrapping a fundamentally crap API of X11 for graphics in this century.

GTK 4 changed that, and it now wraps a Wayland model of API. That drastically simplified GDK, which is why I could write a macOS backend in a couple of weeks.

It also completely changed how we draw. We no longer do immediate mode style (in the form of Cairo) and instead do a retained mode of draw commands. That allows for lots of new things you just couldn't do before with the old drawing model. It will also allow us to do a lot more fun things in the future (like threaded/tiled renderers).

The APIs all over the place were simplified and focused. I can't imagine writing an application the size of GNOME Builder again with anything less than GTK 4.

Hell, last GNOME cycle I rewrote Sysprof from scratch in a couple months, and it's become my co-pilot every day.

Re:GTK: Introducing Graphics Offload by kaba0 on Saturday November 18, 2023 @02:06PM (#38323151)
Thanks for the comment and for your work!
Re:GTK: Introducing Graphics Offload by DonHopkins on Saturday November 18, 2023 @12:24PM (#38321925)
Thank you for the correction, it's a relief! That's nice work.

I'm sorry, I misinterpreted the paragraph in the article saying "exports" as meaning that it exports the pixels from GPU memory to CPU memory, not just passing a reference like GL_TEXTURE_EXTERNAL_OES and IOSurface does.

>GTK has already been using dmabufs since 4.0: When composing a frame, GTK translates all the render nodes (typically several for each widget) into GL commands, sends those to the GPU, and mesa then exports the resulting texture as a dmabuf and attaches it to our Wayland surface.

Perhaps I'd have been less confused if it said "passes a reference to the resulting texture in GPU memory" instead of "exports the resulting texture", because "exports" sounds expensive to me.

Out of curiosity about the big picture, are dmabufs a Linux thing that's independent of OpenGL, or independent of the device driver, or build on top of GL_TEXTURE_EXTERNAL_OES, or is GL_TEXTURE_EXTERNAL_OES/SurfaceTexture just an Android or OpenGL ES thing that's an alternative to dmabufs in Linux? Do they work without any dependencies on X or Wayland or OpenGL, I hope? (Since pytorch doesn't use OpenGL.)

https://source.android.com/docs/core/graphics/arch-st

One practical non-gui use case I have for passing references to GPU textures between processes on Linux is pytorch: I'd like to be able to decompress video in one process or docker container on a cloud instance with an NVidia accelerator, and then pass zero-copy references to the resulting frames into another process (or even two -- each frame of video needs to be run through two different vision models) in another docker container running pytorch, sharing and multitasking the same GPU, possibly sending handles through a shared local file system or ipc (like how IOSurface uses Mach messages to magically send handles, or using unix domain sockets or ZeroMQ or something like that), but I don't know if it's something that's supported at the Linux operating system level (ubuntu), or if I'd have to drop down to the NVidia driver level to do that.

NVidia has some nice GPU video decompressor libraries, but they don't necessarily play well with pytorch in the same process, so I'd like to run them (or possibly ffmpeg) in a different process, but on the same GPU. Is it even possible, or am I barking up the wrong tree?

It would be ideal if ffmpeg had a built-in "headless" way to perform accelerated video decompression and push out GPU texture handles to other processes somehow, instead of rendering itself or writing pixels to files or touching CPU memory.

Re:GTK: Introducing Graphics Offload by audidude on Saturday November 18, 2023 @01:42PM (#38322902)
> Out of curiosity about the big picture, are dmabufs a Linux thing that's independent of OpenGL, or independent of the device driver,

They are independent of the graphics subsystem altogether (although that is where they got their start, afaik). Your webcam also uses DMABUF. So if you want to display your webcam from a GTK 4 application, this GtkGraphicsOffload will help you take that DMABUF from your camera (which may not be mappable on CPU memory, but can DMA pass to your GPU), and display it in a GTK application. It could either be composited on the GPU, or mapped directly to scanout if the right conditions are met.

I wrote a library recently (libmks) and found the culprits in Qemu/VirGL/virtio_gpu that were preventing passing a DMABUF from inside a guest VM to the host. That stuff is all fixed now so theoretically you could even have a webcam in a VM which then uses a GTK 4 application to render with VirGL and the compositor submit the scene to the host OS which itself can set the planes correctly to get the same performance as if it were in the host OS.

> I'd like to be able to decompress video in one process or docker container on a cloud instance with an NVidia accelerator, and then pass zero-copy references

If you want this stuff with NVidia, and you're a customer, I highly suggest you tell your NVidia representative this. Getting them to use DMABUF in a fashion that can be used from other sub-systems would be fantastic.

But at it's core, if you were using Mesa and open drivers for some particular piece of hardware, yes it's capable of working given the right conditions.

Re:GTK: Introducing Graphics Offload by play_ac on Saturday November 18, 2023 @09:07AM (#38319740)
No, that isn't allowing exposure to kernel graphics buffers. That's allowing clients to draw to the main framebuffer with no acceleration at all. If you're memory mapping pixels into user space and drawing with the CPU then you're necessarily leaving kernel space. Around the same time X11 had an extension called DGA that did the same thing. It was removed because it doesn't work correctly when you have hardware acceleration.

So the optimization only makes sense for a machine like yours with no native drivers. With any kind of GPU acceleration it will actually make things much slower. GTK doesn't do this because it would only be useful for that kind of machine running around 25 years ago.

Re:GTK: Introducing Graphics Offload by DonHopkins on Saturday November 18, 2023 @09:15AM (#38319827)
Shared kernel graphics buffers in main or memory mapped framebuffer device memory are one thing (common in the early 80's, i.e. 1982 SunView using /dev/fb), but does it expose modern shared GPU texture buffers to multiple processes, which are a whole other ball game, and orders of magnitude more efficient, by not requiring ping-ponging pixels back and forth between the CPU and GPU when drawing and compositing or crossing process boundaries?
Re:GTK: Introducing Graphics Offload by AshamedCaptain on Saturday November 18, 2023 @10:26AM (#38320581)
> With any kind of GPU acceleration it will actually make things much slower. GTK doesn't do this because it would only be useful for that kind of machine running around 25 years ago.

Precisely one of the points of TFA is be able to use the "25 year old" hardware overlay support whenever possible (instead of the GPU) in order to save power, like Android (and classic Xv) does.

Re:GTK: Introducing Graphics Offload by ris on Saturday November 18, 2023 @10:37AM (#38320674)
> After 25 years, GTK joins the party …

I mean.. shall we start a list of ways in which BeOS/Haiku have yet to "join the party" that the linux desktop has managed?

Juggling the various needs of one hell of a lot more users across a lot more platforms, with a lot more API-consuming apps to keep working on systems which are designed with a lot more component independence is a much harder problem to solve.

Re:GTK: Introducing Graphics Offload by CyberDildonics on Saturday November 18, 2023 @01:30PM (#38322780)
Are you mixing up number of users with technical sophistication?
Re:GTK: Introducing Graphics Offload by pengaru on Saturday November 18, 2023 @11:47AM (#38321440)
Wasn't this one of the main security issues with the BeOS architecture?

It's not the same thing as what's being done here via Wayland. BeOS is more YOLO style direct access of the framebuffer contents, without solving any of the hard problems (I don't think it was really possible to do properly using the available hardware at the time).