News for hackers. Stuff that matters.
Re:Leveraging Rust and the GPU to render user interfaces at ... by yason on Thursday March 09, 2023 @04:45AM (#35079732)
The bottleneck of UI is not the rendering. A measly 60 fps is plenty fast for UI that feels immediate. We had this in the 90's with software rendering, you don't need a GPU for that today.

What causes user interfaces to hick up is that it's too easy to do stuff in the main UI thread. First it doesn't matter but stuff does accumulate, and eventually the UI begins to freeze briefly for example, after you press a button. The user interface gets intermingled with the program logic, and the execution of the program will visibly relay its operations to the user.

It would be very much possible to keep the user interface running in a thread, dedicated to a single CPU on a priority task, updating at vsync rate as soon as there are dirty areas in the window, merely sending UI events to the processing thread, and doing absolutely nothing more. This is closer to how games work: the rendering thread does rendering and there are other, more slow-paced threads running physics, simulation, and game logic at a suitable pace. With games it's obvious because rendering is hard and it needs to be fast so anything else that might slow down rendering must be moved away but UIs shouldn't be any different. An instantly reacting UI feels natural to a human, one that takes its time to act will slow down the brain.

But you don't need a GPU for that.

Re:Leveraging Rust and the GPU to render user interfaces at ... by DeathArrow on Thursday March 09, 2023 @05:07AM (#35079841)
I think the bottleneck comes from updating each UI element instead updating them in batches and updating elements that don't need to be updated.
Re:Leveraging Rust and the GPU to render user interfaces at ... by kaba0 on Thursday March 09, 2023 @05:37AM (#35079982)
That’s just retained mode GUIs calculating what got damaged and only updating those. That’s how most GUI frameworks work since many decades.
Re:Leveraging Rust and the GPU to render user interfaces at ... by eptcyka on Thursday March 09, 2023 @06:03AM (#35080121)
It's significantly cheaper to render a fullscreen window on the GPU than it is on the CPU if you're running at 2560x1600 or 4K. To push that many pixels, you have to send significant amounts of data (framebuffers) at 60 FPS to the GPU anyway - which is no small feat and is bound to eat into the battery. It's just more efficient to run on the GPU.
Re:Leveraging Rust and the GPU to render user interfaces at ... by seanalltogether on Thursday March 09, 2023 @06:04AM (#35080131)
Separating the logic thread from the render thread doesn't end up being the silver bullet to performance. If I press the letter 'a' on my keyboard and my IDE suddenly stalls on a bunch of code hinting logic, that new state is still not going to make it to the render thread for a couple frames. As long as the render thread is dependent on state coming in from the logic thread, the stalls still propogate.

You could have some wins around scrolling and any animations that you can commit ahead of time

Re:Leveraging Rust and the GPU to render user interfaces at ... by tracker1 on Thursday March 09, 2023 @01:24PM (#35084729)
That's a big reason why VS Code pushed the extensions (for code completion logic, etc) to another process. Where VS proper was in-thread... man, some web projects in VS are just painful.

I do think that they will need to do something similar with a higher level language support for extensions. Could probably piggy-back the Deno work for using V8/JS/TS for extensions similar, maybe even compatible with VS Code extensions. It's a massive user space, but seeing the core feature set in another editor could be really nice.

For me the killer feature of VS Code is really the integrated terminal. Not having to switch windows, or leave the app to run a quick command is really big imo.

Re:Leveraging Rust and the GPU to render user interfaces at ... by ben-schaaf on Thursday March 09, 2023 @06:43AM (#35080316)
Unfortunately this isn't true anymore when you get to very high resolutions like 8k. Just clearing the framebuffer of an 8k display at 60hz requires ~6GB/s; about 1/10th of the theoretical memory bandwidth of modern desktop processors. Add in compositing for multiple windows, text rendering and none of that likely being multi-threaded it's pretty clear a CPU has no chance of keeping up.
Re:Leveraging Rust and the GPU to render user interfaces at ... by fxtentacle on Thursday March 09, 2023 @08:29AM (#35080942)
You are correct, but so is the comment you replied to.

The latency that matters for a text editor is how long it takes for a new keypress to show up. And that's usually a 32x32 pixel area or less.

Re:Leveraging Rust and the GPU to render user interfaces at ... by gizmo on Thursday March 09, 2023 @09:01AM (#35081222)
That's not sufficient, though. You want the 0.1% case where you press undo a couple of times and big edit operations get reversed to be smooth. You have to hit your frame target when a lot is happening, the individual key press case is easy.

It's just like a video game. A consistent 60fps is much better than an average frame rate of 120fps that drops to 15fps when the shooting starts and things get blown up. You spend all the time optimizing for the worst case where all your caches get invalidated at the same time.

Re:Leveraging Rust and the GPU to render user interfaces at ... by fxtentacle on Thursday March 09, 2023 @01:04PM (#35084434)
In a video game, you'd just blur the shit out of everything else while you grant yourself a 2nd frame to re-render the rest of the content [1]. Or you do temporal remapping like the Oculus does if you rotate your head too quickly.

[1] https://developer.nvidia.com/blog/nvidia-vrss-2-dynamic-fove...

Re:Leveraging Rust and the GPU to render user interfaces at ... by ben-schaaf on Thursday March 09, 2023 @09:23AM (#35081443)
The other latency that matters is how long it takes to draw each frame when scrolling through the code, which is usually most if not all of the framebuffer.
Re:Leveraging Rust and the GPU to render user interfaces at ... by fxtentacle on Thursday March 09, 2023 @01:06PM (#35084457)
Agree. In that case, you need to cache the result of your CPU rendering in a GPU texture so that you can smoothly scroll it. But you can probably still get away with drawing all UI components on the CPU if you add a little bit of caching.
Re:Leveraging Rust and the GPU to render user interfaces at ... by jcelerier on Thursday March 09, 2023 @09:04AM (#35081250)
But when you do software rendering you don't need to redraw the entire frame buffer, just the parts that changed, which can be very small even at 8k - e.g. the box of a checkbox being switched
Re:Leveraging Rust and the GPU to render user interfaces at ... by ben-schaaf on Thursday March 09, 2023 @09:20AM (#35081411)
Scrolling requires redrawing most if not the whole framebuffer.
Re:Leveraging Rust and the GPU to render user interfaces at ... by HideousKojima on Thursday March 09, 2023 @10:56AM (#35082585)
With scrolling couldn't you just shift the existing framebuffer up/down and only draw the newly scrolled-to parts?
Re:Leveraging Rust and the GPU to render user interfaces at ... by kevingadd on Thursday March 09, 2023 @12:40PM (#35084072)
The GPU buffering model doesn't usually let you read the previous frame like that these days, so you would need a scratch surface. That increases your bandwidth and memory requirements, so shifting doesn't end up being much faster if it's faster at all.
Re:Leveraging Rust and the GPU to render user interfaces at ... by TylerE on Thursday March 09, 2023 @03:31PM (#35086263)
I suspect there are enough edge cases (semi-transparent things jumps to mind) that it's unlikely to be lossless, anyway.
Re:Leveraging Rust and the GPU to render user interfaces at ... by kllrnohj on Thursday March 09, 2023 @11:29AM (#35083075)
You don't need to redraw the entire framebuffer with GPU rendering, either. It's perfectly possible to do small, partial updates.

See for example https://registry.khronos.org/EGL/extensions/EXT/EGL_EXT_buff...

And https://registry.khronos.org/EGL/extensions/KHR/EGL_KHR_part...

Re:Leveraging Rust and the GPU to render user interfaces at ... by CyberDildonics on Thursday March 09, 2023 @10:57AM (#35082614)
You aren't talking about the same thing they are and you aren't talking about a scenario that exists. 8k is still exotic and no one is driving any desktop off of a video buffer going from main memory through a CPU.
Re:Leveraging Rust and the GPU to render user interfaces at ... by ben-schaaf on Friday March 10, 2023 @04:51PM (#35101625)
We've literally got an 8k screen in the office and 4k is becoming increasingly common. Not only is this a scenario that exists it's one I've personally experienced and fixed.
Re:Leveraging Rust and the GPU to render user interfaces at ... by CyberDildonics on Friday March 10, 2023 @07:23PM (#35103390)
Your 8k screen is run purely off of the CPU drawing to a frame buffer?
Re:Leveraging Rust and the GPU to render user interfaces at ... by zodester on Thursday March 09, 2023 @07:25AM (#35080514)
React Native follows this pattern by moving most JS processing off the main thread allowing scrolling and other input to happen without blocking for a response from the JS VM. However this does end up causing a lot of problems with text input and gestures as now you have a sync issue between the threads, if you get caught processing a bunch of stuff in the JS thread the app may appear to be responsive with scrolling but nothing happens in response to button taps or text insertion. It is the only way RN was going to work on lower end hardware though so probably is the right solution if you assume running react everywhere is a good idea.
Re:Leveraging Rust and the GPU to render user interfaces at ... by mike_hearn on Thursday March 09, 2023 @08:43AM (#35081063)
Yes, making GUIs responsive isn't as simple as just "don't run stuff on the UI thread". There are good reasons to run stuff there even if you're going to hang up the app for a brief period, namely, the user won't see partial/incorrect updates like non-syntax highlighted text or incorrectly clipped shapes, and - especially important - it means you can't end up with invalid GUI states like the user pressing a button that does X and then immediately pressing another button that does the opposite of X, where you should have disabled the other button but didn't get to it in time. Web apps have this sort of problem if they aren't using enough JS, and it can cause all kinds of weird bugs and errors.

The reality is that moving things off the UI thread is always a calculated engineering decision. It can be worth it, but it complicates the code and should only be done if there's a realistic chance of the user noticing.

Re:Leveraging Rust and the GPU to render user interfaces at ... by Traubenfuchs on Thursday March 09, 2023 @10:54AM (#35082566)
Proper development of multithreaded desktop apps and not blocking the UI thread appears to be a lost art. I remember I was drawing all the threads on a person wide piece of paper on the wall when I was working on my first commercial winforms application. It's not exactly hard, but requires basic understanding for UX, threading and the platforms you are working with.

Nowadays I still regularly see applications with (temporarily) frozen windows and I just don't understand how that's possible. When I was developing my winforms apps, anything that would do more than perfectly predictable UI manipulations was run on a background thread (in a task), would be forbidden from starting twice at the same time and only updated the UI when done.

Re:Leveraging Rust and the GPU to render user interfaces at ... by HideousKojima on Thursday March 09, 2023 @10:58AM (#35082632)
In .NET Core/.NET 5+ I think it even defaults to having awaited Tasks run on a different thread. So long as you're using async/await properly it's almost impossible to screw up (no need to worry about SynchronizationContext etc.), yet I still see tons of examples of people who simply don't understand threading, async/await, etc. and screw it all up
Re:Leveraging Rust and the GPU to render user interfaces at ... by Traubenfuchs on Thursday March 09, 2023 @11:23AM (#35082980)
As I don't like async/await (read: I probably don't properly understand it) I just added two extension methods to Control that took a lambda for updating UI stuff (one for Invoke, one for BeginInvoke).

Probably kind of like this: https://stackoverflow.com/a/36107907/2306536

I haven't developed C# for almost a decade, so I probably didn't even have access to async/await back then.

Re:Leveraging Rust and the GPU to render user interfaces at ... by HideousKojima on Thursday March 09, 2023 @11:35AM (#35083144)
Even if you don't really understand async/await in C#, you can do it mostly right by following just a few rules:

1) Make anything that accesses a resource over the network, in a database, or in the filesystem async. So basically any time you make an HTTP request, execute a SQL query, or open a file etc. Otherwise you should generally stick to synchronous since async code is actually slightly slower than synchronous code. An exception to this might be if you're doing some heavy computation that's going to take a while and don't want it blocking the UI thread

2) Async is viral. If you have a method that calls an async method, you need to make the calling method viral too. You need to then make its own calling method async as well all the way back to Main()

3) Never use async void as a return type, with the sole notable exception to this rule being event handlers. Use async Task instead as the return type (or async Task<T> if it returns a value)

4) Always call async code with the "await SomeAsyncFunction()" syntax. Calling it other ways like "SomeAsyncFunction().Result" will eliminate most of the benefits of using async in the first place since it is blocking, can cause deadlocks in some cases, etc.

Follow those 4 rules and you don't really even need to understand how async works under the hood, though personally I'd recommend learning it as it will help you a lot when you run into issues with edge cases.

Re:Leveraging Rust and the GPU to render user interfaces at ... by kllrnohj on Thursday March 09, 2023 @11:42AM (#35083251)
> A measly 60 fps is plenty fast for UI that feels immediate. We had this in the 90's with software rendering, you don't need a GPU for that today.

No we didn't. We had nothing close to that in the 90s.

Typing was responsive, but that's not anywhere close to 60fps and a very small region to update to boot (120wpm = 600 keys/minute = 10 keys/s, or 10fps). Scrolling benefits from 60fps+, but in the 90s scrolling jumped by lines at a time because it couldn't do anything better.

You need a GPU to keep up with smooth scrolling at modern resolutions. But this also shouldn't be a surprise. What the article talks about is bog standard stuff for all current UI toolkits. It's what mobile platforms like Android have been doing for a decade now.

Re:Leveraging Rust and the GPU to render user interfaces at ... by TylerE on Thursday March 09, 2023 @03:32PM (#35086274)
60 is the new 30.

Was amazed at how good my latest iPhone feels with 120hz scrolling. It's like magic.

Re:Leveraging Rust and the GPU to render user interfaces at ... by MaxikCZ on Friday March 10, 2023 @01:39PM (#35098908)
I thought moving to 144 Hz monitor made sense just for gaming, but now on 60 Hz monitor just moving the mouse on desktop feels as if the PC is struggling