You don't know that. For one thing, their silicon costs are going to be relatively cheap. It's an old reliable, 14nm process, and compared to even Google's TPU this is a relatively simple chip. For another they _could_ be putting all that silicon to a good use, and by all indications they are. Because there's far less local memory movement, and weights are distributed throughout the system, even this 14nm system could be energy efficient. 9/10ths of all power in a conventional system does not go towards compute - it's wasted in moving data back and forth. This is especially bad in transformers, which, because of their size, largely defeat the memory hierarchies the architects worked so hard to perfect. IOW, all your caches are useless and you're unnecessarily wasting 90% of your energy while also getting worse latency and worse throughput (due to memory bus bandwidth constraints). Oops. These folks seem to be offering something that nobody else does - a feasible, proven way to get out of jail free. I wish them all the success they can get, because all the other currently available architectures are largely unsuitable for high throughput transformer inference, and they work in spite, instead of because, of their design.

Peak H100 power consumption is 700W. Average power consumption of the groq card (from their own website) is 240W. With 576 chips it just doesn’t look good. How much is that millisecond perf gain worth it to end users?

That said I think their arch is super interesting. I just think that demo was way too hype when the actual system is pretty impractical.

So? They aren't performing the same computation. You can't compare the two. What you can compare is power draw at an equivalent tokens/sec on the same model. But you don't have that number.

I’m just estimating here from public numbers. My point is the power consumption could be 100W on that workload and the groq chip could be 1k, both ridiculously optimistic. The whole system is still crazy expensive. H100s will not have latency as fast, but in terms of concurrent users and TCO, I really don’t think groq will be worth it. You could probably get the same concurrent users and throughput with like 8 h100s. Latency won’t be as good, but price could be much lower.

Why would they want to run it themselves if the TCO didn’t work out

Because they rather operate at a loss with high revenue rather than have 0 revenue and loss?

I thought that was par for the course these days.

Operate at a loss. Get a big valuation. Cash out.