> I really hope we'll soon move to a compute-only way of handling graphics

Not happening anytime soon. The industry has lined up pretty solidly behind Khronos/Vulkan.

I know it won't happen overnight, but since it's already possible to do software rasterization for small triangles faster than hardware, having a graphics API framework starts losing its purpose. After all, we want to have the detail of small triangles anyway. Just let us draw to the screen in CUDA without the need for OpenGL/Vulkan interop, and I believe we'll soon see a shift to serious compute-based real-time rendering.

Basically, instead of graphics being a framework, I want graphics to be a straightforward library you include and use in your CUDA/HIP/SYCL/OpenCL code.

> let us draw to the screen in CUDA without the need for OpenGL/Vulkan interop

how would that work? like GPU frameworks would just be compute (like cuda) and some small component of it would just allow to write the end result to a buffer which would be displayed or smth?

Exactly. Things like that already work with a workaround: You can use Cuda-OpenGL interop to expose an OpenGL framebuffer in CUDA, then you can simply write into that framebuffer from your CUDA kernel, and afterwards you get back to OpenGL to display it on screen. Just directly integrate that functionality in CUDA by providing a CUDA native framebuffer and a present(buffer) or buffer swap functionality.

The industry targeting Android and GNU/Linux devices, that is.

>. The industry has lined up pretty solidly behind Khronos/Vulkan.

What industry? Gaming industry is mostly using DirectX or Gnm/PSSL.

Only if you take a very narrow view of the gaming industry that only includes high end consoles and PCs.

The Nintendo Switch and PCs support OpenGL and Vulkan. But those are both dwarfed by mobile gaming which I think makes up more than half of the entire gaming market, and both iPhone and Android support OpenGL. Although I don't know if the majority of mobile games use it or not.

While switch supports OpenGL and Vulkan, in practice very little games use it. The native API on Switch is a proprietary API called NVN.

OpenGL on Apple is on life support to support existing software, Apple platforms are all in on Metal now.

The only platforms where OpenGL and Vulkan are first class citizens is Linux, Android and Windows (barely). And Vulkan is still second fiddle to DirectX on Windows.

All fair points. However, I was taking issue specifically with the claim that the "Gaming industry is mostly using DirectX or Gnm/PSSL".

It would be hard to quantify exactly how much of it is using OpenGL, as a percentage.

However, for a very rough upper bound I quickly looked up these:

* Mobile gaming apparently makes 52% of the entire gaming market in 2023

* Android is 78% of mobile gaming

That could mean OpenGL makes up to 40% of the entire gaming market just from Android.

Although I have no idea what percentage of Android games use OpenGL and wouldn't know how to look that up. I also don't know how accurate those percentages are. I got them from brief Google queries and got both the 52% and 78% figures from the featured snippet.

But even if that 40% is much too high, we still need to add switch and PC games and I would find a figure of 30% of the gaming industry using openGL to be believable. That would put OpenGL as the largest API which I didn't expect, even as a possibility.

Just to reiterate on this point: Pretty much every Vulkan game on Windows forgoes using Vulkan Swapchains in favor of presenting from compute and using native DXGI Swapchains. If you are shipping a production quality game or application, you really want to use the platforms native APIs to get the full capabilities.

Interesting, like which games? I know RDR2 uses normal Vulkan swapchains.

What percentage of applications *natively* target Vulkan?

Imho Vulkan is the least used of all the APIs if you disregard it as a compatibility layer

Isn't nanite hardware rasterization with software shading? Like using the GPU to draw triangle/cluster ID #s into the FB and then shading those?

It uses the rasterization HW for large triangles and uses a software rasterizer in compute for small triangles.

The HW rasterizer is not that efficient if the triangles are tiny, which is the case for nanite.

Not really trivial if you have to support any input set of triangles and don't know much about them. Software rasterizion exploits things like localized chunks of triangles, but the hardware rasterizer does not know about that in advance. Also, these kinds of software rasterization algorithms add limitations, which aren't much of an issue with your own rendering pipeline that specifically knows how to deal with these limitations and works with them.

Part of Nanite is software rasterization by rendering triangles with 64 bit atomics. You can simply draw the closest fragment of a triangle to screen via atomicMin(framebuffer[pixelID], (depth << 32) | triangleData). Not sure right now if they are using a visibility buffer, but if they do than "triangleData" would be the triangle index, otherwise it could be the actual shaded color.

For small triangles, they use a software rasterizer into the V-buffer. Obviously for 1px triangles since they don't want to waste quad overdraw, but I think they found it's faster up to 12px/tri or so on AMD.

> The overhead of vulkan is absolutely insane (and not warranted, in my opinion)

Would you mind expanding on this?

I meant the development/learning overhead. With Vulkan you can do incredible low-level optimizations to squeeze every last bit of performance out of your 3D application, but because you are basically mandated to do it that way, you have a very harsh learning curve and need lots of code for the simplest tasks. I'd rather prefer approaches that make the common things that everyone wants to do easy (draw your first simple scenes), and then optionally gives you all the features to sqeeze out performance where you really need to. Because at least for me, I don't work with massive scenes with millions of instances of thousands of different objects. I do real-time graphics research, mostly with compute, and I'd just like to present the triangles I've created or the framebuffers I created via compute (like shadertoy).

I'll readily admit, I'm neither smart nor patient enough for Vulkan so I quickly gave up and learned CUDA instead, because it was way easier to write a naive software-rasterizer for triangles in CUDA, than it was to combine a compute shader and a vertex+fragment shader in Vulkan. I'm just rendering a single buffer with ~100k compute-generated triangles, and learning Vulkan for that just wasn't worth it.

Well, then there's OpenGL and DirectX.

Yeah, but OpenGL doesn't get updates anymore. My timeline goes like: I needed pointers(and pointer casting) for my compute shaders so I checked the corresponding GLSL extension, which was only available in Vulkan so I tried switching from OpenGL to Vulkan. After a week I gave up - the pointer/Buffer reference extension did not look promising anyway - and I tried out CUDA instead. That's when I found out that CUDA is the greatest shit ever. That's what I want graphics programming to be like. Since then I just render all the triangles and lines in CUDA, because it easily handles hundreds of thousands of them in real-time with a naive, unoptimized software-rasterizer, and that's all I need. In addition to the billion points you can also render in real-time in CUDA with atomics.

> That's when I found out that CUDA is the greatest shit ever. That's what I want graphics programming to be like.

Something which requires you to buy new hardware from a specific brand, and load an out-of-tree binary-only module on your kernel? That's not what I want graphics programming to be like.

The Vulkan API might be clunkier (I don't know, I haven't looked at the CUDA API, since I don't have the required hardware), but at least it can work everywhere.

It's the exact reason why I've avoided CUDA for years, but I hit a dead end with OpenGL and Vulkan, and CUDA happened to be a fantastic, easy and fast solution. Of course I don't want graphics programming to be NVIDIA-only, but I want it to be like CUDA, just for all platforms.

Except it doesn't work everywhere, far from it.

I'm confused by what you mean here by "software-rasterizer" when you are compiling it to run on the GPU. What do you think the graphics driver is doing when you program it with OpenGL? It's doing more or less the same thing under the hood.

I guess you can technically call that software rasterization now that GPUs are very programmable. But it's not how the word is usually used.

GPUs have dedicated hardware for rasterisation. Using compute shaders to do it would be wasteful.

Not necessarily. I'm working on an application that does exactly that, as we are rendering spheres. It ends up being much more efficient to rasterize the sphere in a combination of vertex and fragment shaders than to instantiate triangles.

Nanite renders small triangles faster by avoiding the dedicated hardware and using compute, instead.

Micro-triangles are a special case and Nanite still uses the hardware for more reasonably sized triangles.

"Turns out we can beat the hardware with triangles much bigger than expected, far past micropoly. We software rasterize any clusters whos triangles are less than 32 pixels long." - https://advances.realtimerendering.com/s2021/Karis_Nanite_SI...

It's not just micro-triangles, it works for triangles that span multiple pixels. And that's not a special case, that's the standard nowadays, except for games targeting very low-end devices.

The dedicated hardware is nice for general-purpose support for arbitrary triangle-soups. But if you structure triangles a certain way and have a certain amount of density (which you want for modern games), you can specifically optimize for that and beat the general-purpose hardware rasterizer.

And webgpu.

Sure, the implementations today are thin layers over D3D12, Vulkan and Mantel. But it is a proper API, cross platform and without the implementation overhead.

Next time I need to draw some graphics I will definitely use WebGPU instead of OpenGL or Vulkan.

PS. Don't let the name fool you. It is a proper render API, Mozilla has written their implementation in rust and Google in C++.

Would not be surprised if WebGPU do get native support by the drivers in the future.

> The overhead of vulkan is absolutely insane

Overstatement of the year award candidate.

Vulkan and OpenGL both already support mesh shaders, which is a compute-oriented alternative to the traditional rasterization pipeline.

> Vulkan and OpenGL both already support mesh shaders, which is a compute-oriented alternative to the traditional rasterization pipeline.

Mesh shaders are a step in the right direction, but they are still embedded in all that unnecessary Vulkan fluff. I would want these things in CUDA because it does the opposite approach of Vulkan - it makes the common things easy, and the hard/powerful things optional. Just let me draw things directly in CUDA, and maybe give access to the hardware rasterizer via a CUDA call.

> CUDA [...] makes the common things easy

Only if the common thing does not include running on hardware of more than one vendor. Which is a pretty common requirement for any consumer software.

You're missing the point. CUDA is easy but also powerful, and it shows there is no reason that other APIs need to be hard and cumbersome.

> Overstatement of the year award candidate.

I think he meant development overhead, not performance overhead.

Yes, sorry for the confusion. I'd just rather have an API where the common things are easy, and the super powerful low-level optimizations are optional.

There were several pushes to produce a "sane defaults" library that made vulkan a lot less verbose.

OpenGL is exactly what you said, "an API where the common things are easy, and the super powerful low-level optimizations are optional."

except the easy common things are very slow, and the optimizations which are basically required become a careful dance of understanding the underlying hardware interface anyway (without any real documentation) and figuring out how to map that onto the high-level interface.

You could always just use a library that simplifies Vulkan or keep some skeleton apps around. When you're past all of the verbose set up, the actual meat of it isn't too bad.

What happened is that Khronos Group banked on third parties writing OpenGL-esque "middleware libraries" that bridged the gap between the hopelessly complex Vulkan API and people who want to write simple 3D graphics. Unfortunately, Khronos learnt the hard way that you can't just expect people to do work for you without a sufficient incentive, since after all, OpenGL still exists for anyone who wishes to have an easy API for graphics programming, and anyone who doesn't want to do that just uses an engine or high-level graphics API.

Of course I'm not saying that these libraries don't exist, but they haven't taken off in the way they expected

> who wishes to have an easy API for graphics programming, and anyone who doesn't want to do that just uses an engine or high-level graphics API.

I'm guessing this niche is just pretty small in commercial sense; most people probable end up just using UE/Unity/... as their graphics API.

I agree, so I think Khronos expected the open source community to fill in the gaps

* ... a compute-oriented continuation of the practice where they take the average of Nvidia and AMD's implementations, and then force the developer to deal with the overhead of harmonizing the details without going insane.

GPU compute is awful, almost as bad as GPU rendering.

>but I really hope we'll soon move to a compute-only way of handling graphics.

Would you have the time to expand on this thought a bit? I am curious. Thanks!!

Software rendering. In parallel on the GPU.

I mean, about 5 years ago there were many folks that were trying to do raytracing using GPU compute rather than the current methods, it was essentially treating the CPU as a giant parallel software rendered. The results were pretty good even then.

* * *

I don't get what that means. You mean configuring a shader to render pixels?

How do you think OpenGL/Vulkan works under the hood? It's been a very long time since fixed function pipelines.

The standard rendering pipeline is still fairly fixed and mandates shaders for vertices and fragments. With software rasterization, there are no more vertex or fragment shader, only compute. And you don't use the hardware rasterization units - you rasterize the triangles yourself and write the results to the framebuffers with atomic-min/max operations. You decide for yourself how and when you compute the shading in your compute shader. This can be multiple times faster for small triangles, and 10-100 times faster for points. And once you do things that way, there isn't much point for graphics APIs anymore - everything is just buffers and functions that process them.

Why is there a pipeline at all? Give me a framebuffer, a place to upload some CUDA or equivalent, and some dumb data pipe between the host program and GPU program.