News for hackers. Stuff that matters.

A comment on Groq Raises $640M to Meet Soaring Demand for Fast AI Inference. Parent comment

Re:Groq Raises $640M to Meet Soaring Demand for Fast AI Infe... by zaptrem on Monday August 05, 2024 @01:45PM (#41163541)
I don't understand why all of their hubub is about tokens per second (which relies on a lot more than just hardware) instead of FLOPs/memory bandwidth/etc.
[ Parent | on HN ]
Re:Groq Raises $640M to Meet Soaring Demand for Fast AI Infe... by elorant on Monday August 05, 2024 @04:08PM (#41165045)
They provide those numbers too. Memory bandwidth is at whooping 80 TBs/second.
[ Parent | on HN ]
Re:Groq Raises $640M to Meet Soaring Demand for Fast AI Infe... by consumer451 on Monday August 05, 2024 @01:49PM (#41163588)
They do make their own hardware, but of note is that they stopped selling hardware and instead pivoted to only selling API access, 4 months ago. Hence no need to describe anything aside from tokens/sec?

https://news.ycombinator.com/item?id=39964590

[ Parent | on HN ]
Re:Groq Raises $640M to Meet Soaring Demand for Fast AI Infe... by mrtksn on Monday August 05, 2024 @01:58PM (#41163673)
Maybe because that's how its measured in the LLM community? It's all about tokens per second.
[ Parent | on HN ]