Groq Raises $640M to Meet Soaring Demand for Fast AI Inference

Re:Groq Raises $640M to Meet Soaring Demand for Fast AI Infe... by mrtksn on Monday August 05, 2024 @01:18PM (#41163292)

AFAIK they are a hardware company producing special chips for processing AI tasks. Their response time is incredibly fast, like 500t-900t/s for LLMs. They're not just a company that buys in Nvidia cards for serving as a backend.

In other words, they are competing with Nvidia, not the million startups using Nvidia.

[ Parent | on HN ]

Re:Groq Raises $640M to Meet Soaring Demand for Fast AI Infe... by zaptrem on Monday August 05, 2024 @01:45PM (#41163541)

I don't understand why all of their hubub is about tokens per second (which relies on a lot more than just hardware) instead of FLOPs/memory bandwidth/etc.

[ Parent | on HN ]

Re:Groq Raises $640M to Meet Soaring Demand for Fast AI Infe... by elorant on Monday August 05, 2024 @04:08PM (#41165045)

They provide those numbers too. Memory bandwidth is at whooping 80 TBs/second.

[ Parent | on HN ]

Re:Groq Raises $640M to Meet Soaring Demand for Fast AI Infe... by consumer451 on Monday August 05, 2024 @01:49PM (#41163588)

They do make their own hardware, but of note is that they stopped selling hardware and instead pivoted to only selling API access, 4 months ago. Hence no need to describe anything aside from tokens/sec?

https://news.ycombinator.com/item?id=39964590

[ Parent | on HN ]

Re:Groq Raises $640M to Meet Soaring Demand for Fast AI Infe... by mrtksn on Monday August 05, 2024 @01:58PM (#41163673)

Maybe because that's how its measured in the LLM community? It's all about tokens per second.

[ Parent | on HN ]