News for hackers. Stuff that matters.
Re:Nvidia pursues $30B custom chip opportunity with new unit by tmaly on Saturday February 10, 2024 @01:50PM (#39329063)
I would love to see a consumer graphics card with 128GB VRAM

Would be nice to be able to work with some of the larger open source LLM models.

Re:Nvidia pursues $30B custom chip opportunity with new unit by karolist on Saturday February 10, 2024 @03:31PM (#39329925)
Not exactly what you've asked but Mac Studio exists, with 192GB at that.
Re:Nvidia pursues $30B custom chip opportunity with new unit by rrrix1 on Sunday February 11, 2024 @03:03PM (#39338114)
Calling a Mac Studio equipped with 192GB of RAM "consumer grade" is a big stretch.

"Consumer ${thing}" to me is somewhere =< $3000.

Re:Nvidia pursues $30B custom chip opportunity with new unit by karolist on Sunday February 11, 2024 @04:28PM (#39338927)
True, but you get a full general compute machine for under $6k, after a few years you can sell it for 3-4k and upgrade to the latest one. Personally I think it's the best thing to happen to local LLMs, by accident. I have an M1 Max 64GB and I'm blown away every day by these local models. Apple didn't plan for it, but it happened by accident due to unified memory with godzillion throughput having integrated silicon.
Re:Nvidia pursues $30B custom chip opportunity with new unit by Aurornis on Saturday February 10, 2024 @03:27PM (#39329880)
Unfortunately, as soon as you make a card with specifications that make it great at enterprise-grade tasks, it will be bought in mass quantities by people building out data centers. This pushes the price up, as we’ve already seen.

So labeling it “consumer” doesn’t really mean much. They’ve tried to enforce the distinction with EULAs before, but that doesn’t work well.

Re:Nvidia pursues $30B custom chip opportunity with new unit by sydd on Saturday February 10, 2024 @03:37PM (#39329972)
> it will be bought in mass quantities by people building out data centers.

Isn't this good? The production at scale effects will kick in, lowering the price and supply will meet demand after some hiccups.

Re:Nvidia pursues $30B custom chip opportunity with new unit by wmf on Saturday February 10, 2024 @04:09PM (#39330245)
Nvidia doesn't want the price to be lower; that's why they won't make this card.
Re:Nvidia pursues $30B custom chip opportunity with new unit by Aurornis on Saturday February 10, 2024 @03:46PM (#39330042)
If demand drove prices down like that then we’d already have cheap cards available.

Demand puts upward pressure on prices.

Supply is already maxed out and growing as fast as possible.

Re:Nvidia pursues $30B custom chip opportunity with new unit by piva00 on Saturday February 10, 2024 @04:32PM (#39330422)
> Isn't this good? The production at scale effects will kick in, lowering the price and supply will meet demand after some hiccups.

You didn't account for the "hiccups", which can vary from 5-20 years until competition catches up, longer than the life of many companies. In spherical cows worlds of economics that would be just a hiccup.

Re:Nvidia pursues $30B custom chip opportunity with new unit by LoganDark on Saturday February 10, 2024 @07:58PM (#39331753)
> Supply is already maxed out and growing as fast as possible.

No, it's not anywhere near maxed out anymore. The chip shortage has been over for a while; now there is too much supply.

But that's not the issue. The issue is that Nvidia really wants to force these enterprise customers to pay extremely high prices. This is why they are so afraid to give any more VRAM to the consumer chips, because if they were at all suitable for VRAM-heavy workloads, every HPC company would buy a couple $2,000 consumer cards in place of each $15,000 datacenter card. That'd lose Nvidia something like 70% which they would find entirely unacceptable.

Re:Nvidia pursues $30B custom chip opportunity with new unit by blackoil on Saturday February 10, 2024 @09:37PM (#39332224)
> chip shortage has been over for a while; now there is too much supply.

Is that at 10-40nm or at cutting edge 3-5nm?

If there is a surplus supply, why aren't they making more H800, there demand is more than supply.

Re:Nvidia pursues $30B custom chip opportunity with new unit by LoganDark on Saturday February 10, 2024 @10:11PM (#39332360)
By "too much supply" I mean that once the chip shortage ended, they tried to take advantage of that and produced far too many 30-series cards. When the mining crash happened, they went surprised pikachu face because then they couldn't easily sell off the rest of their inventory.

Not sure if they still have surplus.

Re:Nvidia pursues $30B custom chip opportunity with new unit by brucethemoose2 on Saturday February 10, 2024 @03:35PM (#39329952)
We are getting that (in early 2025?) With AMD Strix Halo.

40 CUs, 256 bit LPDDR5X, 16 CPU cores. Or so the rumors say.

Re:Nvidia pursues $30B custom chip opportunity with new unit by datameta on Saturday February 10, 2024 @01:56PM (#39329128)
I think that goal is fine and good, but I would rather see huge investments toward in-memory compute like ReRAM and such. If we bridge the efficiency advancements of TinyML with the leap of LLM abilities, perhaps we can start on the road of not being limited by the impact of training on climate.
Re:Nvidia pursues $30B custom chip opportunity with new unit by epistasis on Saturday February 10, 2024 @04:07PM (#39330229)
Agree with the comment, but riffing on the last few words:

Climate is pretty much my #1 concern about the world, but LLM use of energy is really really far down on the list of important actions for climate.

First and foremost are removing roadblocks for deploying existing technologies for clean energy, and speeding up the necessary supporting infrastructure such as transmission and market policies for choosing cheapest possible solutions (over the objections of dinosaur execs that choose last century's solutions). Then the big hard to decarbonize parts of industry like cement and steel, as well as deploying electrolyzers to get ammonia fertilizer production switched over to carbon neutral production rather than from fossil-generated hydrogen.

Reducing energy consumption is important for advancing AI in general, but ultimately all its energy consumption will be from clean energy sources anyway, and the switch that needs to happen is that switch in energy sources. Reducing energy use by 2x or 10x is not good enough, we must change the sources fundamentally.

Re:Nvidia pursues $30B custom chip opportunity with new unit by asicsarecool on Saturday February 10, 2024 @03:15PM (#39329778)
Why the fuck was this downvoted.

Very occasionally I get the feeling HN is entering the /. phase

Re:Nvidia pursues $30B custom chip opportunity with new unit by wmf on Saturday February 10, 2024 @03:39PM (#39329987)
I didn't downvote it but in-memory compute is crackpot and alternative memory tech is really crackpot. It's not going to happen and it's ridiculous to propose it on the same level as GPUs with more RAM.
Re:Nvidia pursues $30B custom chip opportunity with new unit by zozbot234 on Saturday February 10, 2024 @06:07PM (#39331086)
GPU's are already closer to "in-memory compute" compared to CPU's. It's just taking the existing pattern of NUMA (non-uniform memory access) to a greater extent, as a principled approach to the so-called 'Von Neumann bottleneck'.
Re:Nvidia pursues $30B custom chip opportunity with new unit by imtringued on Saturday February 10, 2024 @08:01PM (#39331777)
In-memory compute is very easy, you just don't have to fall for the pipe dream of using the same process for both the memory and the compute. All you have to do is follow a package on package strategy like we already do with smartphones. A Raspberry PI 5 gets 25GB/s memory bandwidth and it only has a single DRAM chip if I recall correctly.

So if you had a DIMM with 16 of these chips, you would already be on the same bandwidth as HBM. 96 DIMMs and you get 40TB/s memory bandwidth.

Re:Nvidia pursues $30B custom chip opportunity with new unit by datameta on Sunday February 11, 2024 @02:02PM (#39337517)
Can you explain further why you believe it isn't worth exploring? Is it that you can't imagine how it would scale to the level of today's high performance compute hardware?

Just ten years back, squeezing ML models onto microcontrollers sounded completely insane, given their tight memory and power constraints. We've seen NN compilers developed, game-changing techniques like quantization, pruning, and graph-level optimizations pruning. This allowed deployment of ML models in microcontrollers with a newly developed framework like TFLite Micro.

Re:Nvidia pursues $30B custom chip opportunity with new unit by wmf on Sunday February 11, 2024 @02:40PM (#39337905)
These technologies have already been explored and they failed every time. It's throwing good money after bad.

Also, speculative basic research isn't comparable to adding more RAM to a graphics card. You can't substitute one for the other.

Re:Nvidia pursues $30B custom chip opportunity with new unit by amelius on Saturday February 10, 2024 @02:15PM (#39329292)
That's overkill for any type of graphics application though.

What you want is a DL or parallel compute card, not a graphics card.

They are far more expensive though because compute doesn't sell to the average consumer like graphics does.

Re:Nvidia pursues $30B custom chip opportunity with new unit by dotnet00 on Saturday February 10, 2024 @02:42PM (#39329522)
A modern graphics card already is a parallel compute card. A modern graphics pipeline is mostly compute with only very specific stuff using fixed function functionality.
Re:Nvidia pursues $30B custom chip opportunity with new unit by amelius on Saturday February 10, 2024 @02:50PM (#39329580)
Yes, but that was not the point. The point is that they are sold as graphics cards, not compute cards. Therefore, expect these cards to be good at (and have the memory for) typical graphics operations.
Re:Nvidia pursues $30B custom chip opportunity with new unit by dotnet00 on Saturday February 10, 2024 @11:46PM (#39332721)
I don't entirely agree, especially with the popularity of LLMs nowadays. While graphics are still a big portion of the pitch, compute oriented features are increasingly expected as part of the offering, especially at the higher end of the lineup. If you've bought a 3090/4090, you likely had in mind more computationally oriented tasks compared to if you've bought a 1660 or AMD card.
Re:Nvidia pursues $30B custom chip opportunity with new unit by justinclift on Saturday February 10, 2024 @06:22PM (#39331212)
Are the ray tracing cores in Nvidia cards used for compute?
Re:Nvidia pursues $30B custom chip opportunity with new unit by dotnet00 on Saturday February 10, 2024 @06:34PM (#39331292)
All that the ray tracing cores do is accelerate calculation of BVH intersections. The rest of the tracing (ie the part that gives meaning to the BVH intersection result) is done via compute, with the RT cores being able to be 'invoked' essentially arbitrarily. This is in contrast to the way graphics cards used to be, where you had certain stages for certain things, designed specifically around producing graphics, making the 'graphics' part of the name hold more weight than nowadays.

I'm not sure if you're asking if they're used for a compute application or if you're asking if they can be used from the same execution path as 'compute' code.

Re:Nvidia pursues $30B custom chip opportunity with new unit by zozbot234 on Saturday February 10, 2024 @10:09PM (#39332352)
Seen from first principles, BVH is quite close to some kinds of hierarchical clustering. The ray intersection part can be understood in turn as a kind of normalization, or projection to the unit sphere. So depending on how much hardware is actually reserved to this task on-chip, there might be ways of making it somewhat useful in the ML context.
Re:Nvidia pursues $30B custom chip opportunity with new unit by justinclift on Sunday February 11, 2024 @02:38AM (#39333259)
Just asking if those cores tend to sit unused when "standard" compute jobs are run on the cards. :)
Re:Nvidia pursues $30B custom chip opportunity with new unit by sigmoid10 on Saturday February 10, 2024 @02:20PM (#39329343)
Meh, that's like saying 640kb of RAM is more than anyone would ever need. Demand follows hardware development, which in turn accelerates demand. I'm sure game developers would easily find a way to use 128GB of VRAM if it was commonly available in their target market.
Re:Nvidia pursues $30B custom chip opportunity with new unit by godelski on Saturday February 10, 2024 @02:56PM (#39329633)
This isn't an argument against giving users this much VRAM, but I'm pretty sure they'd just drop the whole game into VRAM and call it a day instead of actually optimizing.
Re:Nvidia pursues $30B custom chip opportunity with new unit by fbdab103 on Saturday February 10, 2024 @04:10PM (#39330259)
Isn't that exactly how things improve? When you can reduce the cognitive load on bookkeeping, you can accomplish more productive work.

I can produce a featureful webapp in Python-Django solo because I do not have to worry about optimally filling registers and CPU cache. As you cut deeper to the hardware limitations, you have to be significantly more cognizant of writing to the hardware constraints than solving the business problem.

Taken to the extreme, we have Electron applications consuming multi-GB of RAM, but it does expand the universe of possibilities.

Re:Nvidia pursues $30B custom chip opportunity with new unit by LoganDark on Saturday February 10, 2024 @08:07PM (#39331799)
> Isn't that exactly how things improve? When you can reduce the cognitive load on bookkeeping, you can accomplish more productive work.

What are you talking about? Sure you can appear to be more productive if you are reckless and wasteful of resources, and adding more resources means there is more that you can afford to waste.

But for the past decade or so we have been seeing what happens if you teach every software developer that it's okay to waste resources: resources get wasted.

Back when you couldn't afford to waste resources, you'd see extremely skilled optimizations and extremely clever hacks to get advanced software to run on extremely slow and primitive CPUs. But ever since computing power increased by a couple orders of magnitude, what happens? The exact same software implemented by today's developers would more than compensate for the increased computing power by being a couple orders of magnitude less efficient.

Sure it expands the universe of possibilities. I do want more resources, I really do. But "reducing the cognitive load on bookkeeping" is making software get slower faster than hardware gets more powerful. Which is absolute insanity to me.

Re:Nvidia pursues $30B custom chip opportunity with new unit by fbdab103 on Saturday February 10, 2024 @09:46PM (#39332259)
Think of all the software which does not exist if we are still writing in assembly because we are constrained by the hardware. Python, RPGMaker, Roblox, PICO-8, etc are no longer possible because they are too wasteful. Having bounty enables art to be made which is impossible if creators are bit twiddling.

Do I bemoan that my 2000s era super computer squanders resources to load 20 kb of text? Sure, but it is also amazing the things that are possible because tools are not restricted to the priesthood of assembly programmers.

Re:Nvidia pursues $30B custom chip opportunity with new unit by LoganDark on Saturday February 10, 2024 @10:07PM (#39332342)
> Think of all the software which does not exist if we are still writing in assembly because we are constrained by the hardware.

There is an entire spectrum between "640k ought to be enough for anybody" and "this app needs its own fully independent copy of an entire browser engine just to display a couple screens of text".

About Electron: Operating systems provide their own system-level webviews. Use those. Tauri uses those; Tauri apps are absolutely tiny and use very little RAM, because the system-level webview doesn't have to duplicate the overhead of an entire browser engine for every single app that uses it. It typically only has to duplicate the overhead of a single page.

But that is for apps that even need a webview. For apps that don't, you don't have to drop all the way down to assembly to pick something like WinForms. Even WinUI 3 isn't always terrible, at least compared to webviews.

Hell, I wonder if even Python + wxPython would be more efficient than using a webview. Honestly, none of the software you mentioned is "too wasteful", or at least those aren't what I was referring to with my original comment.

Have you heard about Microsoft Teams, and how it can take up to 10 seconds just to display the splash screen, even on a decently fast computer? I'd like to see you try to tell me that it would have literally killed them to optimize it a bit more. Just think back to, say, the IRC clients from back in the Windows 7 days where you wouldn't even have to blink before they finished opening up.

Re:Nvidia pursues $30B custom chip opportunity with new unit by godelski on Sunday February 11, 2024 @02:33AM (#39333239)
Another example is just how few simple optimizations I see in code now. Something like

   `while i < someFunction():`
Instead of

   ```
   criteria = someFunction()
   while i < criteria:
   ```
I've seen simple lines like this compound in complexity as someFunction() gets more complex or as the loop increases. There's a billion variations on the idea but I think it is because people aren't thinking about how someFunction() is getting called every iteration. I do think every programmer should have some basic optimization skills just because these types of things would be nonobvious otherwise. Because frankly, this kind of optimization is going to have no real difference in a toy program or even in many isolated test cases but it can easily add meaningful latency to a program. Definitely more egregious examples but just trying to think of something that is extremely low hanging fruit.

The real issue comes when the user is doing things more than just your program...

Re:Nvidia pursues $30B custom chip opportunity with new unit by summm on Sunday February 11, 2024 @06:15AM (#39334227)
wouldn't any compiler optimize this anyway?
Re:Nvidia pursues $30B custom chip opportunity with new unit by godelski on Sunday February 11, 2024 @02:15PM (#39337658)
No, and a lot of python code is written these days.
Re:Nvidia pursues $30B custom chip opportunity with new unit by godelski on Sunday February 11, 2024 @02:26AM (#39333207)
I have this theory that if you remove too much friction then things get worse. Because when there's friction it makes people cognizant of limitations. But when there's not enough friction it becomes easy to sweep aside or ignore. You can be too reliant on your metrics. Idk if this actually makes sense but maybe there's a nugget somewhere in the idea.
Re:Nvidia pursues $30B custom chip opportunity with new unit by godelski on Sunday February 11, 2024 @02:22AM (#39333199)
> Isn't that exactly how things improve?

There's a joke "20 years of software progress has completely undone the 20 years of hardware progress."

Essentially the joke is about how people don't optimize software like they used to. There is some tongue in cheek though because there are more platforms and variations that we have to account for these days but there is also truth that there is some laziness. I mean I have a laundry app on my phone that takes 5s to load each time because it redraws the home screen while it is trying to connect to the network and find the machines but won't use the bluetooth or nfc chips that both my phone and the machines have.

I do think there is the case to move fast and break things, but at some point you got to slow down and fix things. There should be cycles. Don't forget that debt has interest and that interest compounds. If you have enough debt you'll be "moving fast" but being fast in molasses isn't fast. It's kinda why a lot of software have just declared bankruptcy and started over (e.g. "rewriting in rust"). We should definitely encourage people to hire a mixture of those that move fast and those that move slow. There's harmony there but I think we've forgotten that.

Re:Nvidia pursues $30B custom chip opportunity with new unit by LoganDark on Saturday February 10, 2024 @10:34PM (#39332455)
cough Apple Silicon cough cough

Unified memory is honestly a really, really good idea.

Supposedly, if you have professional Apple hardware experts hand-optimize a machine-learning model, then Apple's most powerful M3 chips can compete against, say, a typical/unoptimized PyTorch implementation of the same model on a 4090.

Except the M-series chips can currently have up to 8 times the memory of the 4090, and they don't have to worry about shuffling things in and out of VRAM. The GPU can just work with any of the data that the CPU already has.

Re:Nvidia pursues $30B custom chip opportunity with new unit by godelski on Sunday February 11, 2024 @02:35AM (#39333246)
I'd be hard pressed to buy a comparison to a 4090. But yeah, things would be better if they didn't "dump everything into VRAM"
Re:Nvidia pursues $30B custom chip opportunity with new unit by LoganDark on Sunday February 11, 2024 @07:31AM (#39334585)
They're comparing incredibly hand-optimized MLX inference on the Mac to off-the-shelf stuff on the 4090. So I'd believe it just because the 4090's being given an artificial disadvantage, and the Mac is being given meticulous hand-optimization by the foremost experts on the very platform it runs on.

In other words, the Mac is being given a huge optimization advantage compared to the 4090. And it's roughly competitive.

Even that is no mean feat.

Source: https://appleinsider.com/articles/23/12/13/apple-silicon-m3-...

Re:Nvidia pursues $30B custom chip opportunity with new unit by amelius on Saturday February 10, 2024 @02:52PM (#39329602)
Turns out that demand for graphics memory got stuck at a point where compute is still hungry for more.

That may certainly change but it doesn't help compute much, today.

Re:Nvidia pursues $30B custom chip opportunity with new unit by LoganDark on Saturday February 10, 2024 @10:47PM (#39332497)
It's supply that's down - Nvidia is strictly enforcing their artificial segmentation where the jump from 24GB to 40GB of VRAM multiplies the price by around 5 (!).

They know that if you need a 40GB card, you probably can't use a 24GB card at all because it'd run out of VRAM. They charge such insane prices for those higher capacity cards, because they know that they can make you pay thousands upon thousands for every tiny scrap of extra VRAM, and you will have literally no choice but to pay that price because you need the VRAM.

Demand isn't low, demand is actually so high that they don't care if lowly consumers want VRAM, they already have enterprise customers that need it so badly that they'll pay prices higher than most consumers would ever imagine.

And that's why they're so stingy with VRAM on the consumer cards. They need to be careful not to accidentally make them useful in the datacenter, so they can ensure that their enterprise customers continue to be forced to pay those extortionary prices.

You see, it's all about the money, and it always will be. Capitalism, baby!

Re:Nvidia pursues $30B custom chip opportunity with new unit by amelius on Sunday February 11, 2024 @11:39AM (#39336160)
You need a special license to run nVidia cards in the datacenter, so this rules out consumer gaming cards already.
Re:Nvidia pursues $30B custom chip opportunity with new unit by LoganDark on Thursday February 15, 2024 @07:11PM (#39391201)
There are already ways to get around this. For example, renting compute from people who aren't in datacenters. Which is already a thing: https://vast.ai
Re:Nvidia pursues $30B custom chip opportunity with new unit by Takennickname on Saturday February 10, 2024 @02:49PM (#39329565)
Can you imagine the download sizes?
Re:Nvidia pursues $30B custom chip opportunity with new unit by late2part on Saturday February 10, 2024 @02:48PM (#39329557)
To be fair, 640kb is more than anyone really needs. It's just far, far less than we want.
Re:Nvidia pursues $30B custom chip opportunity with new unit by blackoil on Saturday February 10, 2024 @09:42PM (#39332246)
As a side effect we are seeing lot of investment/innovation in 2-13B models. Considering price of 4090, 128GB GPU will be >6000 which practically no one can afford.
Re:Nvidia pursues $30B custom chip opportunity with new unit by elabajaba on Sunday February 11, 2024 @01:47PM (#39337397)
This isn't possible currently unless you use HBM, which is significantly more expensive both to actually buy the HBM, and to package it (and all the capacity for packaging HBM interposers is already used for data center hardware).

The most VRAM you could have on a consumer GPU today is 48GB, and that'd be on a 4090 or 7900xtx with clamshell VRAM (which increases cost and makes cooling significantly harder due to putting GDDR6 chips on the back of the GPU where there aren't any fans).

To calculate how much VRAM is possible, you just need to divide the GPU's bus width by 32 (or 64 for Samsung's new weird double capacity but double bit width) and multiply that by the largest GDDR capacity currently available (16Gbit).

As for why GPUs don't increase their bus width, there have been GPUs with 512bit busses in the past, but it makes it quite a bit more expensive (more vram chips, more traces to run, might require more/heavier PCB layers) and increases power draw.