News for hackers. Stuff that matters.
Re:Groqchat by badFEengineer on Friday December 22, 2023 @05:46PM (#38739479)
This was surprisingly fast, 276.27 T/s (although Llama 2 70B is noticeably worse than GPT-4 turbo). I'm actually curious if there's good benchmarks for inference tokens per second- I imagine it's a bit different for throughput vs. single inference optimization, but curious if there's an analysis somewhere on this

edit: I re-ran the same prompt on perplexity llama-2-70b and getting 59 tokens per sec there

Re:Groqchat by andygeorge on Friday December 22, 2023 @06:28PM (#38739910)
fast but wrong/gibberish
Re:Groqchat by razorguymania on Friday December 22, 2023 @06:31PM (#38739936)
Its using vanilla llama-2 from Meta with no fine tuning. The point here is the speed and responsiveness of the underlying HW and SW.
Re:Groqchat by andygeorge on Friday December 22, 2023 @06:59PM (#38740211)
Do you work there? Just curious
Re:Groqchat by razorguymania on Friday December 22, 2023 @07:08PM (#38740273)
yes
Re:Groqchat by andygeorge on Friday December 22, 2023 @07:13PM (#38740304)
Thanks!
Re:Groqchat by andygeorge on Friday December 22, 2023 @06:46PM (#38740077)
ah Llama 2 70B, no wonder
Re:Groqchat by chihuahua on Friday December 22, 2023 @11:19PM (#38741597)
But if the quality of the response is poor, it's irrelevant that it was generated quickly. If it was using different data to generate higher quality responses, would that not slow it down?
Re:Groqchat by tome on Saturday December 23, 2023 @03:45AM (#38742664)
nomel gave a good answer in a different thread

> This is not about the model, it’s about the relative speed improvement from the hardware, with this model as a demo.

To compare apples to apples look at the tokens per second of other systems running Llama 2 70B 4096. We're by far the fastest!

https://news.ycombinator.com/item?id=38742466