News for hackers. Stuff that matters.
Re:Nvidia to join Dow Jones Industrial Average, replacing Intel by dagmx on Friday November 01, 2024 @10:36PM (#42023649)
LLMs absolutely did not emerge because of NVIDIA. you’re the one imho who is mistaking correlation with causality.

The first transformer models were developed at Google. NVIDIA were the card du jour for accelerating it in the years since and have contributed research too, but your statement goes way too far

Re:Nvidia to join Dow Jones Industrial Average, replacing Intel by llm_trw on Friday November 01, 2024 @11:19PM (#42023812)
I was around the space before Alexnet came out. Without NVidia the last 15 years of AI would not have happened.

Deep learning was only possible because you could do it on NVidia cards with Cuda without having to use the big machines in the basement.

Trying to convince anyone that neural networks could be useful in 2009 was impossible - I got a grant declined and my PhD supervisor told me to drop the useless tech and focus on something better like support vector machines.

Re:Nvidia to join Dow Jones Industrial Average, replacing Intel by dagmx on Friday November 01, 2024 @11:29PM (#42023853)
I was around then too and people forget that AMD also had GPU compute APIs.

The difference is AMD killed theirs to use OpenCL and NVIDIA kept CUDA around as well.

Re:Nvidia to join Dow Jones Industrial Average, replacing Intel by llm_trw on Friday November 01, 2024 @11:36PM (#42023881)
Just like how ROCm is supposed to be competitive today and it isn't unless you have an army of grad students to look after your data center cards.

I tried using AMD Stream and it lacked documentation, debugging information and most of the tools needed to get anything done without a large team of experts. NVidia by comparison could - and was - used by single grad students on their franken-stations which we build out of gaming GPUs.

The less we talk about the disaster that the move to opencl was the better.

Re:Nvidia to join Dow Jones Industrial Average, replacing Intel by dagmx on Friday November 01, 2024 @11:46PM (#42023929)
I agree ROCm was a mess and only recently is even really usable.

That said ROCm is quite a recent thing borne out of acknowledging that OpenCL 2 was a disaster.

OpenCL 1 had a reasonable shot and was gaining adoption but 2 scuppered it.

Re:Nvidia to join Dow Jones Industrial Average, replacing Intel by llm_trw on Saturday November 02, 2024 @01:33AM (#42024265)
Did you ever train anything past mnist on AMD? I try every five years or so and promise myself never again each and every time.
Re:Nvidia to join Dow Jones Industrial Average, replacing Intel by dagmx on Saturday November 02, 2024 @02:17AM (#42024396)
Not with Rocm since I’ve moved my personal stack to NVIDIA (for rendering) and Macs for day to day use.

I did write quite a bit of OpenCL prior to that on Intel/AMd/NVIDIA, both for training and for general rendering though, and did some work with Stream before then.

Re:Nvidia to join Dow Jones Industrial Average, replacing Intel by llm_trw on Saturday November 02, 2024 @02:46AM (#42024506)
Was it OpenCL1? That's the only one I hadn't tried out for AMD GPUs. Everything else I have and can say with absolute certainty that you spend more time fighting the hardware than you did writing code.

Cuda by comparison JustWorks^tm.

Re:Nvidia to join Dow Jones Industrial Average, replacing Intel by dagmx on Saturday November 02, 2024 @02:57AM (#42024547)
Both 1 and 2. I haven’t done much with 3 as OpenCL is effectively a dead api at this point.

1 was definitely a lot easier to work with than 2. CUDA is easier than both but I don’t think I hit anything I could do in CUDA that I couldn’t do in OpenCL, though CUDA of course had a larger ecosystem of existing libraries.

Re:Nvidia to join Dow Jones Industrial Average, replacing Intel by llm_trw on Saturday November 02, 2024 @03:14AM (#42024604)
Damn, I missed the best one.

Thanks for the trip down memory lane.

Re:Nvidia to join Dow Jones Industrial Average, replacing Intel by aledalgrande on Saturday November 02, 2024 @12:53PM (#42027534)
I was not a PhD, but I studied both around that period in university and can confirm that neural networks were only seen as a kind of theoretical plaything. All the libraries were pushing SVMs.
Re:Nvidia to join Dow Jones Industrial Average, replacing Intel by BobbyJo on Friday November 01, 2024 @10:42PM (#42023675)
I think their point is that easily accessible hardware capable of supporting the research is the reason the research has come as far as it has, and I would tend to agree. At the very least, GPUs keeping the PCI ecosystem going has played a major role in allowing the specialized accelerator market to flourish.
Re:Nvidia to join Dow Jones Industrial Average, replacing Intel by dagmx on Friday November 01, 2024 @10:50PM (#42023705)
But that could apply to any of the GPU manufacturers. CUDA made for an easier ecosystem but if it didn’t exist it would have been any of the other APIs.

The first transformer models didn’t even use CUDA and CUDA didn’t have mass ecosystem inroads till years later.

I’m not trying to downplay NVIDIA but they specifically mentioned cause and effect, and then said it was because of NVIDIA.

Re:Nvidia to join Dow Jones Industrial Average, replacing Intel by KaoruAoiShiho on Friday November 01, 2024 @10:58PM (#42023741)
First transformer != LLM. Imagine a world where you had to use AMD or CPUs, there would be no AlexNet, there would be no LLMs. Nvidia seeded universities with gifted hardware accelerators for a over a decade. Nvidia built the foundations for modern ML on which transformer lies, it's just one step on a long road to LLMs.
Re:Nvidia to join Dow Jones Industrial Average, replacing Intel by dagmx on Friday November 01, 2024 @11:27PM (#42023845)
AMD had GPU compute APIs at the time as well. They also used to contribute GPUs (albeit in much smaller quantities). They just ended up killing them in favor of OpenCL which then withered on the vine.

NVIDIA absolutely contributed to the foundation but they are not the foundation alone.

Alexnet was great research but they could have done the same on other vendors at the time too. The hardware didn’t exist in a vacuum.

Re:Nvidia to join Dow Jones Industrial Average, replacing Intel by KaoruAoiShiho on Saturday November 02, 2024 @12:15AM (#42024030)
Alexnet could not have been done on AMD, nope.
Re:Nvidia to join Dow Jones Industrial Average, replacing Intel by dagmx on Saturday November 02, 2024 @12:36AM (#42024101)
Please, this is just revisionism. There’s nothing inherent to AlexNet that relied on NVIDIA hardware or even software. It was just what happened to be available and most straightforward at the time.

To say it wouldn’t have been possible on AMD is ludicrous and there is a pattern to your comments where you dismiss any other companies efforts or capabilities, but are quite happy to lay all the laurels on NVIDIA.

The reality is that multiple companies and individuals got us to where we are, and multiple products could have done the same. That's not to take away from NVIDIA's success, it's well earned, but if you took them out of the equation, there's nothing that would have prevented the tech existing.

Re:Nvidia to join Dow Jones Industrial Average, replacing Intel by hnlmorg on Saturday November 02, 2024 @04:44AM (#42024972)
The crux of the issue is this:

> It was just what happened to be available and most straightforward at the time

AMD made better hardware for a while and people wanted OpenCL to succeed. The reason why nvidia became dominant was because their competitors simply weren’t good enough for general purpose parallel compute.

Would AI still have happened without CUDA? Almost certainly. However nvidia still had a massive role in shaping what it looks like today.

Re:Nvidia to join Dow Jones Industrial Average, replacing Intel by EVa5I7bHFq9mnYK on Saturday November 02, 2024 @12:23PM (#42027339)
Many people here are heavily invested in NVDA stock, which certainly clouds the judgement.
Re:Nvidia to join Dow Jones Industrial Average, replacing Intel by dagmx on Saturday November 02, 2024 @04:23PM (#42028986)
That’s my take away too, even though I’m invested myself. The discussion around them has transcended into mythology. Imho part of it is because GPUs themselves are so foreign to many people as a development target and so they’re not really as aware of the landscape
Re:Nvidia to join Dow Jones Industrial Average, replacing Intel by physicsguy on Saturday November 02, 2024 @01:54AM (#42024309)
OpenCL itself is kinda fine, but the libraries never existed because every academic from postdoc up could get a free NVidia card. You literally filled a form out and they sent you it.
Re:Nvidia to join Dow Jones Industrial Average, replacing Intel by dagmx on Saturday November 02, 2024 @02:14AM (#42024383)
I agree the library situation for OpenCL never stood up. Khronos tried at the start but then lost steam.

It’s one of the projects I think the Khronos group mishandled the most unfortunately.

Re:Nvidia to join Dow Jones Industrial Average, replacing Intel by DeathArrow on Saturday November 02, 2024 @04:03AM (#42024804)
Do they have a project that achieved success?
Re:Nvidia to join Dow Jones Industrial Average, replacing Intel by echoangle on Saturday November 02, 2024 @04:17AM (#42024862)
OpenGL, Vulkan, WebGL?
Re:Nvidia to join Dow Jones Industrial Average, replacing Intel by dagmx on Saturday November 02, 2024 @10:48AM (#42026704)
Here’s a list of projects

https://www.khronos.org/

If I were to categorize the successful ones:

glTF, KTX, SPIR-V, OpenGL and its variants, WebGL

People will say Vulkan but it has the same level of adoption as OpenCL, and has the same issue that it competes against vendor specific APIs (DX and Metal) that are just better to use. It’s still used though of course as a translation target but imho that doesn’t qualify it as a success.

OpenCL was and is a failure of grand magnitude. As was colada.

Re:Nvidia to join Dow Jones Industrial Average, replacing Intel by BobbyJo on Saturday November 02, 2024 @12:37AM (#42024102)
> The first transformer models didn’t even use CUDA and CUDA didn’t have mass ecosystem inroads till years later.

I graduated college in 2010 and I took a class taught in CUDA before graduating. CUDA was a primary driver of NN research at the time. Sure, other tools were available, but CUDA allowed people to build and distribute actually useful software which further encouraged the space.

Could things have happened without it? Yeah, for sure, but it would have taken a good deal longer.