AFAIK they are a hardware company producing special chips for processing AI tasks. Their response time is incredibly fast, like 500t-900t/s for LLMs. They're not just a company that buys in Nvidia cards for serving as a backend.

In other words, they are competing with Nvidia, not the million startups using Nvidia.

I don't understand why all of their hubub is about tokens per second (which relies on a lot more than just hardware) instead of FLOPs/memory bandwidth/etc.

They provide those numbers too. Memory bandwidth is at whooping 80 TBs/second.

They do make their own hardware, but of note is that they stopped selling hardware and instead pivoted to only selling API access, 4 months ago. Hence no need to describe anything aside from tokens/sec?

https://news.ycombinator.com/item?id=39964590

Maybe because that's how its measured in the LLM community? It's all about tokens per second.

$640M of fundraising seems notable enough for this level of HN attention.

It's a hardware company and justified to be on HN imo. They're doing interesting stuff.