News for hackers. Stuff that matters.
Re:A look at Apple's technical approach to AI including core... by zarzavat on Friday June 14, 2024 @05:35AM (#40679156)
NVIDIA are not selling shovels, they are selling shovel-making machines to the shovel-sellers (the companies making foundation models).

When the market is saturated with shovel-making machines, what then?

Re:A look at Apple's technical approach to AI including core... by esmy on Friday June 14, 2024 @07:18AM (#40679700)
The last time I remember this happening, the shovel-making machines lost a lot of value and got amortised out to the second hand market. When the prevailing coin still mined on GPUs switched algorithms the bottom of GPU prices fell out on eBay.

I expect at some point a) the training and the inference will move to even more specialised hardware and the current existing silicon go to either hobbyists or some other market that requires the current cards b) the way in which we do AI may fundamentally change in ways we can't predict, causing the required hardware to also change (move to FPGA or favouring some other aspect than VRAM)

In both scenarios NVIDIA isn't the one who win, because the sudden influx lifts a secondary market nv don't profit from directly.

I personally don't think they'll be able to sustain their current valuation past the current AI rush, and will compact back down to somewhere their levels pre-chatgpt.

That said I'm just a guy on the Internet who wouldn't mind some beefy ex-datacentre gear if I could make it work for video editing workloads.

Re:A look at Apple's technical approach to AI including core... by pbronez on Friday June 14, 2024 @07:33AM (#40679782)
Heh I am so ready to put an A100 in my homelab. Gonna take some intelligent power management though…
Re:A look at Apple's technical approach to AI including core... by xattt on Friday June 14, 2024 @07:40AM (#40679827)
Have you considered running busbars from your closest pole transformer?
Re:A look at Apple's technical approach to AI including core... by pbronez on Friday June 14, 2024 @11:39AM (#40681837)
haha no fire hazards there!

The official specs [0] say the A100 80GB has a 300W TDP. That's pretty doable on standard domestic power... harder if you want to run multiples.

[0] https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Cent...

Re:A look at Apple's technical approach to AI including core... by Gracana on Friday June 14, 2024 @08:27AM (#40680241)
Same here. My eBay watchlist always has a few in it, but the prices just aren't where I need them to be. For now I am focused on inference with a dual 3090 rig, and my next step will be to fit another pair of 3090s, which is the max I can easily support with my AM5 motherboard. What's slowing me down is the fact that I can't run server hardware due to noise, and I can't run an open-air setup because my cats would get into it, so I need to buy or build a case to hold everything.
Re:A look at Apple's technical approach to AI including core... by pbronez on Friday June 14, 2024 @11:43AM (#40681870)
Dug into A100 sale prices using Ebay's Terapeak analytics. I found several recent sales for $2000 or less:

https://www.ebay.com/itm/335392812272?nordt=true

https://www.ebay.com/itm/335408349837?nordt=true

https://www.ebay.com/itm/335414821299?nordt=true

Too good to be true? Perhaps. For $2000 I might trust Ebay's buyer protections...

Re:A look at Apple's technical approach to AI including core... by Gracana on Wednesday June 26, 2024 @03:21PM (#40803530)
Wow! Yeah, I would have taken a shot at that. The seller seems legitimate, judging by their other listings.
Re:A look at Apple's technical approach to AI including core... by ants_everywhere on Friday June 14, 2024 @09:24AM (#40680709)
This is totally not a space I understand. But it seems like there are possible threats to NVIDIA, such as

(1) MLIR, ROCm, or some other tooling that reduces CUDA's moat,

(2) AMD attracting investment to go after NVIDA

(3) ARM-based GPUs or accelerators gaining traction among cloud companies that have huge fleets of AI devices and also have the money to devote to custom chips

Can anyone who understands the industry explain why those threats (or similar ones) aren't a major issue for NVIDIA?

Re:A look at Apple's technical approach to AI including core... by gmays on Saturday June 15, 2024 @09:41PM (#40694056)
Not an expert, but can give it a shot:

(1) Much development is already moving from CUDA to the LLM, so less of an issue. Nvidia is also doing more work to increase interoperability. Could be an issue I guess, but doesn't seem like it since there's nothing close to CUDA or the ecosystem.

(2) AMD has attracted significant investment looking at appreciation in its market cap, with a PE ration 3X Nvidia's. However, AMD is so far behind in so many ways, I don't believe it is an investment problem, but structural. Nvidia has just been preparing for this for so long it has a temendous head start, not to mention being more focused on this. Remember AMD also competes with Intel, etc.

(3) Hyperscalers already are building their own chips. It seems even Apple used its own chips for Apple Intelligence. It's relatively (which is doing a lot of lifting in this sentence because it's all HARD) not too hard to make custom chips for AI. The hard (near impossible) thing is making the cutting edge chips. And the cutting edge chips are what the OpenAIs of the world demand for training, but releasing the newest best model 1-3 months ahead of a competitors is worth so much.

If anything, I'd say the biggest threat to Nvidia in the next 1-3 years is an issue with TSMC or some new paradigm that makes Nvidia's approach suboptimal.

Re:A look at Apple's technical approach to AI including core... by ants_everywhere on Friday June 21, 2024 @02:38PM (#40752354)
Thanks, that was extremely helpful.

I don't think I understand your point in (1) that it's less of an issue because development is moving to the LLM. I can infer that maybe CUDA isn't a big part of the moat given your other points that the hard part is making cutting edge chips.

Re:A look at Apple's technical approach to AI including core... by gmays on Friday June 21, 2024 @04:39PM (#40753568)
It's just the natural evolution of tech towards higher levels of abstraction. In the beginning most dev was on CUDA because the models had to be built and trained.

But since there are plenty of more advanced models now, the next level is getting built out as more developers start building applications that use the models (e.g. apps using GPT's API).

So where 5 years ago most AI dev was on CUDA, now most is on the LLMs that were built with CUDA to build applications.