Re:Groq Raises $640M to Meet Soaring Demand for Fast AI Infe...
by recursivecaveat on Monday August 05, 2024 @04:24PM
(#41165222)
I don't think that's really a fair comparison. Like if you wanted to sell ball bearings you could spend $50 and start dripping lead into a bucket of water, or you could spend millions on a factory. The latter is definitely the better business model. What maters is quality of result (latency and tokens/s/user) and throughput per watt or capex dollar, not your minimum capex expense so long as you're running at any kind of scale.
[ Parent
| on HN ]
Re:Groq Raises $640M to Meet Soaring Demand for Fast AI Infe...
by elorant on Monday August 05, 2024 @05:00PM
(#41165568)
How is throughput per watt affordable? Each card has a TDP of 275 watts, and a maximum draw of 375. If you try to run a big model like Llama-3 400B you're looking at a cluster with 1,8k cards, and a total power draw of nearly 500KWs per hour.