News for hackers. Stuff that matters.
Re:Triton: Open-Source GPU Programming for Neural Networks by dragontamer on Wednesday July 28, 2021 @01:16PM (#27986289)
AMD's ROCm 4.0 now supports cooperative groups, which is probably one of the last major holdouts for CUDA compatibility.

There's still the 64-wavefront (for AMD CDNA cards) instead of 32-wavefronts (for CUDA). But AMD even has 4x4 half-float matrix multiplication instructions in ROCm (for MI100, the only card that supports the matrix-multiplication / tensor instructions)

---------

I think CUDA vs OpenCL is over. ROCm from AMD has its restrictions, but... it really is easier to program than OpenCL. Its a superior model: having a single language that supports both CPU and GPU code is just easier than switching between C++ and OpenCL (where data-structures can't be shared as easily).

-----------

The main issue with AMD is that they're cutting support for their older cards. The cheapest card you can get that supports ROCm is Vega56 now... otherwise you're basically expected to go for the expensive MI-line (MI50, MI100).

Re:Triton: Open-Source GPU Programming for Neural Networks by meragrin_ on Wednesday July 28, 2021 @02:23PM (#27987534)
> The main issue with AMD is that they're cutting support for their older cards.

No, their main issue is not properly supporting ROCm in general. No Windows support at all? It still feels like they don't know whether they want to continue investing in ROCm long term.

Re:Triton: Open-Source GPU Programming for Neural Networks by dragontamer on Wednesday July 28, 2021 @02:28PM (#27987612)
I'd assume that they're gonna support ROCm as long as the Frontier deployment at Oak Ridge National Labs is up. ORNLs isn't exactly a customer you want to piss off.
Re:Triton: Open-Source GPU Programming for Neural Networks by N1H1L on Wednesday July 28, 2021 @07:42PM (#27991135)
Yes. I am hopeful Frontier will shake things up.
Re:Triton: Open-Source GPU Programming for Neural Networks by zozbot234 on Wednesday July 28, 2021 @01:48PM (#27986872)
The new contest is not CUDA vs. OpenCL but CUDA vs. Vulkan Compute. As support for Vulkan in hardware becomes more widespread, it makes more and more sense to just standardize on it for all workloads. The programming model is quite different between the two (kernels vs. shaders) and OpenCL 2.x has quite a few features that are not in Vulkan, but the latest version of OpenCL has downgraded many of these to extensions.
Re:Triton: Open-Source GPU Programming for Neural Networks by dragontamer on Wednesday July 28, 2021 @01:50PM (#27986925)
CUDA programmers choose CUDA because when you make a struct FooBar{}; in CUDA, it works on both CPU-side and GPU-side.

Vulkan / OpenCL / etc. etc. don't have any data-structure sharing like that with the host code. Its a point of contention that makes anything more complicated than a 3-dimensional array hard to share.

Yeah, Vulkan / OpenCL have all sorts of pointer-sharing arrangements (Shared Virtual Memory) or whatnot. But its difficult to use in practice, because they keep the concepts of "GPU" code separate from "CPU" code.

---------

When you look at these things: such as Triton, you see that people want to unify the CPU-and-GPU code into a single code base. Look and read these Triton examples: they're just Python code, inside of the rest of CPU-Python code.

I think people are realizing that high-level code can flow between the two execution units (CPU or GPU) without changing the high level language. The compiler works hard to generate code for both systems, but its better for the compiler to work rather than the programmer to work on integration.

Re:Triton: Open-Source GPU Programming for Neural Networks by enos_feedler on Wednesday July 28, 2021 @06:06PM (#27990374)
Is it practical for code to flow between these two execution units? In my understanding, the architecture of a GPU and CPU are so fundamentally different that it doesn’t reallly make sense. You do want to have a single data structure representation however.
Re:Triton: Open-Source GPU Programming for Neural Networks by dragontamer on Wednesday July 28, 2021 @06:10PM (#27990415)
> In my understanding, the architecture of a GPU and CPU are so fundamentally different that it doesn’t reallly make sense.

Yes and no. You find surprising bits of code that can be shared.

Read through a high-performance GPU project like "GPU perft" (https://github.com/ankan-ban/perft_gpu/blob/master/chess.h). The "perft" problem is the problem of counting the number of board positions reachable in X moves. (Perft(15) is "how many board positions exist within 15 moves from the start of chess?". Perft(1) is 20, and Perft(2) is 400, then 3+ gets a bit complicated!).

You can see that the fundamentals of bitboard manipulation (using the 64-bit number to represent the 64-squares of a chessboard) remain the same whether or not you're on a CPU or GPU.

In this case, you want the code to be shared between the two sides. Why write the code twice? Both CPUs and GPUs are very good at 64-bit integer manipulation.

------------

The actual search (how to branch off, coalesce results, coordinate threads) is extremely different between CPU and GPU. So of course, you want that code to be written in a CPU-specific, or GPU-specific manner.

But the question of "where can this Bishop move??" (the sliding piece attack subroutine) is identical on CPU or GPU.

Re:Triton: Open-Source GPU Programming for Neural Networks by enos_feedler on Wednesday July 28, 2021 @06:56PM (#27990809)
Sounds similar to business logic shared between and client and server web app. Thanks

I just wonder if this alone is enough to make a unified programming environment the preferred way. On the web we had Meteor.js which tried this approach to unify client and server with javascript. The shared code was of similar types. Meteor never became the preferred way to write web apps. Im wondering if the same is true for GPU + CPU programming.

Re:Triton: Open-Source GPU Programming for Neural Networks by nl on Wednesday July 28, 2021 @09:01PM (#27991601)
This analogy is insufficient to capture what is going on.

In web dev, the client and server are doing fundamentally different things (eg UX vs data storage) and while sharing business logic is interesting it doesn't solve a huge problem.

CPU vs GPU is vaugly analogous to different web browsers on the client in the old days when they were very different. They have different performance characteristics, so shims worked but had performance impact. But they were widely used because it was too hard to do yourself.

Re:Triton: Open-Source GPU Programming for Neural Networks by enos_feedler on Thursday July 29, 2021 @01:36AM (#27993135)
I don’t think this is accurate. CPU and GPU are not trying to execute the same code with different performance characteristics. GPU and CPU are fundamentally doings different things. In fact the analogy for shared business logic works perfectly
Re:Triton: Open-Source GPU Programming for Neural Networks by dragontamer on Thursday July 29, 2021 @11:45AM (#27997880)
The other poster is right. The CPU and GPU are just two execution resources available to the programmer, no different than spinning up a new pthread_create (well... a bit more complicated but you get the gist).

Even in video games, where GPUs do graphics and CPUs do physics, the lines have been blurred with TressFX (hair physics on GPU instead of CPU) or PhysX (collision physics done on GPU instead of CPU).

GPUs are general purpose computers now. You can program them to do whatever you want. So when the programmer is writing code, it pays to experiment a little: maybe run the code on the GPU... see if the GPU really is slower or faster. You might be surprised.

Ditto on the reverse: maybe some bit of GPU code would be better run on the CPU.

--------

This guess-and-check, program-and-profile methodology is the same as any other high-performance code people write. You throw away assumptions and just test the heck out of all your ideas, because you're surprised far more often than not in this field.

As such, having the ability to quickly move code from GPU-side to CPU-side (and vice versa) for these tests is extremely beneficial. You don't really know which device will run your code faster until you actually test it. (Though your instincts get better the more experience you gain)

Re:Triton: Open-Source GPU Programming for Neural Networks by nl on Friday July 30, 2021 @01:00AM (#28004638)
To expand on this:

GPUs are (basically) hardware devices for doing vast numbers of matrix manipulations very very fast in parallel, with limitations on the size of the matrix.

CPUs can do matrix manipulations, with less parallelism.

Things like TressFX uses the tridiagonal matrix algorithm[1] to solve a linear system representing the hair constraints system. This could be implemented on a CPU as well, doing exactly the same thing with the same algorithm. It would run slower though!

[1] https://en.wikipedia.org/wiki/Tridiagonal_matrix_algorithm

[2] Slide 37, https://www.slideshare.net/DevCentralAMD/gs4147-billbilodeau

Re:Triton: Open-Source GPU Programming for Neural Networks by nl on Friday July 30, 2021 @01:02AM (#28004649)
No, they are trying to execute exactly the same code. To quote the parent:

You can see that the fundamentals of bitboard manipulation (using the 64-bit number to represent the 64-squares of a chessboard) remain the same whether or not you're on a CPU or GPU.

In this case, you want the code to be shared between the two sides. Why write the code twice? Both CPUs and GPUs are very good at 64-bit integer manipulation.

The ceremony around co-ordinating threads and memory access is different, but the code that is being run is exactly the same.

Re:Triton: Open-Source GPU Programming for Neural Networks by enos_feedler on Friday July 30, 2021 @06:13PM (#28013066)
The exact same business logic (helper functions), yes of course. My point is the high level algorithm is going to be fundmanetally different. Just like the high level algorithm on the browser (render and handle UI interaction) is different than the fundamental server-side algorithm (render html from database requests), and yet there might be shared helper functions. Does this make sense?
Re:Triton: Open-Source GPU Programming for Neural Networks by nl on Saturday July 31, 2021 @09:03PM (#28022820)
No, the algorithm is exactly the same.

For example, in neural networks the key algorithm is differentiation. That is implemented the same way on both CPUs and GPUs.

Re:Triton: Open-Source GPU Programming for Neural Networks by enos_feedler on Thursday August 05, 2021 @11:58PM (#28083183)
differentiation isn't really the key algorithm. In fact, I believe that happens at compile time, not run-time. So not only is not key, it isn't even happening at run-time.
Re:Triton: Open-Source GPU Programming for Neural Networks by pjmlp on Friday July 30, 2021 @02:31AM (#28005080)
Given the state of Vulkan tooling the contest is already won, NVidia doesn't need to worry about anything.
Re:Triton: Open-Source GPU Programming for Neural Networks by Athas on Wednesday July 28, 2021 @05:05PM (#27989702)
> ROCm from AMD has its restrictions, but... it really is easier to program than OpenCL. Its a superior model: having a single language that supports both CPU and GPU code is just easier than switching between C++ and OpenCL (where data-structures can't be shared as easily).

ROCm isn't a programming language or programming model - it's a collection of tools and libraries. Which language are you specifically referencing here? HIP?

Re:Triton: Open-Source GPU Programming for Neural Networks by dragontamer on Wednesday July 28, 2021 @05:09PM (#27989749)
Yeah, HIP specifically. That's the part of ROCm that is aiming to give a CUDA-like environment.
Re:Triton: Open-Source GPU Programming for Neural Networks by N1H1L on Wednesday July 28, 2021 @07:42PM (#27991129)
Two reasons also. Some of the fastest research supercomputers in the US have been nVIDIA based. Sitting at a national lab, it's laughable how much more support there is from nVIDIA. Getting a DGX box is easy - there is nothing comparable from AMD's side.

And the less said about Intel it's better. Smart, motivated people I know have left Argonne recently because of the Aurora shitshow.

Re:Triton: Open-Source GPU Programming for Neural Networks by bradneuberg on Wednesday July 28, 2021 @11:00PM (#27992338)
Can you comment more on what’s going on at Argonne related to Aurora?
Re:Triton: Open-Source GPU Programming for Neural Networks by N1H1L on Thursday July 29, 2021 @12:06AM (#27992675)
Aurora was supposed to be up and running in early 2018. It seems like that it won't be functional even in 2021. This is by the way, such an years long delay has never happened when it comes to US Leadership Computing Facilities.

Intel, last I heard has written off close to 300 million dollars on Aurora.

Re:Triton: Open-Source GPU Programming for Neural Networks by dragontamer on Thursday July 29, 2021 @01:00PM (#27998651)
Do you know if there's any post-mortems about what went wrong there?

I mean, I have the gist. Intel bet big on Xeon Phi, but that didn't seem to offer enough performance. Intel then switched over to this GPU-strategy (see Intel Xe), but that required them to rework virtually everything from scratch.

No one said anything in my previous paragraph. But its blatantly obvious: the Xeon Phi was being advertised very strongly by Intel up to the point that Aurora's design was reworked. Suddenly, Xeon Phi was cut, and Intel started talking about Xe (including OneAPI, and all sorts of stuff the HPC market cares about). I'm confident enough at reading between the lines that I'm willing to assume this in the absence of hard facts :-)

I'm kind of curious on the play by play, if at all possible. What test showed up that made Argonne National Laboratory decide that the Xeon Phi model wasn't going to work? Was it possible to come to this conclusion any sooner? I realize this sort of stuff is rarely published, but I guess that's what makes me interested in those juicy details.

Re:Triton: Open-Source GPU Programming for Neural Networks by N1H1L on Monday August 02, 2021 @05:05PM (#28042618)
My friends were at more junior levels so I don't know the full details too. However know this, a leadership class computing purchase by a national lab is a very complicated thing. Many people have their hands in the decision making pie - all the way up to Secretary of Energy.

Secondly, DOE has a strategy (mostly rightly in my opinion) of not putting all its' computation eggs in a single companies' basket. Thus national labs compete between themselves as do companies. Notice the cadence of computing purchases: IBM/Nvidia -> Intel/Intel -> AMD/AMD.

Thirdly, Intel had promised that using OneAPI existing GPU optimized simulation codes could be translated to Intel GPUs with minimum effort. The idea had merit, back in 2016-2017. I have a CUDA based simulation code - it's almost a matter of recompiling to Intel, with some minimal effort on my part. That it didn't work out - well hindsight is 20/20

Re:Triton: Open-Source GPU Programming for Neural Networks by anon_tor_12345 on Wednesday July 28, 2021 @09:08PM (#27991653)
E.g. Pavan lol
Re:Triton: Open-Source GPU Programming for Neural Networks by N1H1L on Thursday July 29, 2021 @12:02AM (#27992658)
Balaji is legit. He is an ex Panda student and those guys know their stuff
Re:Triton: Open-Source GPU Programming for Neural Networks by einpoklum on Wednesday July 28, 2021 @04:11PM (#27989119)
> ROCm from AMD has its restrictions, but... it really is easier to program than OpenCL

That really not the point. OpenCL is a standard that - at least in principle - is supposed to be supported on multiple platforms by multiple vendors. ROCm is AMD-only, and even that is questionable since it didn't exist 6 or 7 years ago, and who knows - they might drop it like they changed their earlier focus.

Also, CUDA has a much richer ecosystem than AMD ROCm (I'm sad to say; as I'm not a fan of NVIDIA).

Re:Triton: Open-Source GPU Programming for Neural Networks by dragontamer on Wednesday July 28, 2021 @04:19PM (#27989208)
> OpenCL is a standard that - at least in principle - is supposed to be supported on multiple platforms by multiple vendors.

As was C++AMP (which was actually pretty good IMO as a language). Just because its a standard doesn't mean its going to be used.

OpenCL 2.0 was very poorly implemented: almost no one used any of its advanced features. To the point that OpenCL 3.0 is resetting from OpenCL 1.2.

Only Intel really supported OpenCL 2.1 or OpenCL 2.2. The entire OpenCL 2.x branch for years was squandered with tepid responses from NVidia and AMD (yes, AMD had better OpenCL 2.0 support. But it's debugger didn't work, all of the code was tested on OpenCL 1.2 only. No one cared)

I dare say that OpenCL 2.x was about as "standardized" and respected as C++ AMP. Just because its an open standard doesn't mean that its actually useful. Any serious OpenCL programmer stuck with OpenCL 1.2, including both AMD and NVidia OpenCL programmers.

Re:Triton: Open-Source GPU Programming for Neural Networks by einpoklum on Wednesday July 28, 2021 @06:08PM (#27990398)
Pretty much agree; and I'll add NVIDIA has also made it a point not to support profiling (and probably debugging) OpenCL... to this day.

Still, that doesn't contradict what I said in my earlier post.

There was also OpenCL C++, which I really wanted to see implemented by GPU vendors, especially NVIDIA, and that didn't happen either.

As for C++AMP - did that ever work on NVIDIA hardware?

Re:Triton: Open-Source GPU Programming for Neural Networks by dragontamer on Wednesday July 28, 2021 @07:23PM (#27991013)
C++ AMP compiled into Microsoft's DirectCompute which works on Nvidia (through the video game DirectX stuff)
Re:Triton: Open-Source GPU Programming for Neural Networks by my123 on Wednesday July 28, 2021 @04:13PM (#27989131)
Reminder that ROCm is still not supported on RDNA and RDNA2.