How to make LLMs go fast

Re:How to make LLMs go fast by benchess on Friday December 22, 2023 @03:30PM (#38738306)

If I understand correctly, Groq chips have 220MB SRAM and the next best level is DDR4? How many chips are needed to run Llama2-70B at those speeds?

[ Parent | on HN ]

Re:How to make LLMs go fast by tome on Friday December 22, 2023 @04:53PM (#38739106)

Cool that you know the tech specs of the GroqChip! Yes, that's right, 220 MB of SRAM per chip. I think the demo where we first broke 200 tokens / sec was running on 1 GroqRack, so 64 chips. The live public demo that's currently running at 275 tokens / sec I think might be running on two GroqRacks, so 128 chips. I'm not certain of either of these figures so please don't quote me! But those are the right ball-park.

[ Parent | on HN ]

Re:How to make LLMs go fast by trsohmers on Friday December 22, 2023 @07:27PM (#38740438)

This article from less than a month ago says that it is on 576 chips https://www.nextplatform.com/2023/11/27/groq-says-it-can-dep...

[ Parent | on HN ]

Re:How to make LLMs go fast by tome on Saturday December 23, 2023 @02:59AM (#38742484)

Thanks, looks like you're right and this demo is running on 9 GroqRacks (576 chips). I think we may also have an 8 rack version in progress. We've tried a variety of different configurations to improve performance, which is possible because of the high level of flexibility and configurability of our architecture and compiler tool chain.

[ Parent | on HN ]