News for hackers. Stuff that matters.
Re:Gemini 2.0: our new AI model for the agentic era by menaerus on Wednesday December 11, 2024 @03:33PM (#42392630)
Each GPU costs ~50k. You need at least 8 of them to run mid-sized models. Then you need a server to plug those GPUs into. That's not commodity hardware.
Re:Gemini 2.0: our new AI model for the agentic era by whimsicalism on Wednesday December 11, 2024 @05:48PM (#42393911)
more like ~$16k for 16 3090s. AMD chips can also run these models. The parts are expensive but there is a competitive market in processors that can do LLM inference. Less so in training.
Re:Gemini 2.0: our new AI model for the agentic era by menaerus on Thursday December 12, 2024 @03:56AM (#42397477)
> more like ~$16k for 16 3090s

I don't know where did you get that price from but 1x RTX 3090 is $1,900. 16x is ~$30,000.

> The parts are expensive

Now that we invested ~$30k in GPUs, we only need to find a motherboard that can accommodate 16x pcie4 x16 GPUs, right? And we also need a CPU that can drive that many pcie4 x16 lanes?

Well, none of them exist, not even in the server parts sector let alone client commodity hardware. In any case, you'd need two CPUs so even with this imaginary motherboard we are already entering the server rack design space. And that costs 100's of thousands of $$$.

> but there is a competitive market in processors that can do LLM inference

Nothing but the smallest and smallish models. If that existed then why would you set yourself out building a 16x RTX 3090 machine?

Sorry, but you're just spitting out non-sense.

Re:Gemini 2.0: our new AI model for the agentic era by whimsicalism on Thursday December 12, 2024 @11:57AM (#42400902)
You can just run the GPUs on different nodes, it will still let you run the model. I think the definition of ‘commodity hardware’ in your head is different than the colloquial definition.

GoogleFS was run on ‘commodity hardware’ ie normal disks, of course they were still in a data center but the hardware was still commodity. Whether something is expensive is entirely different from whether it is a commodity, I think you are conflating commodity and consumer.