Re:How to make LLMs go fast
by andy99 on Friday December 22, 2023 @12:23PM
(#38736351)
The speed of transformers is memory bandwidth limited. I think that it's possible to speed them way up with different chip architectures, and I know there are lots of people working on this e.g. https://etched.ai making asics and https://untether.ai making chips will co-located memory and compute cells. So unless some architecture really starts beating transformers badly on language tasks, I think the speed problem is going to disappear as silicon architectures adapt.
[ Parent
| on HN ]
Yup, the company I work for has the world's fastest LLM appliance, based on a hardware design co-locating memory and compute. It requires compilation innovation as well in a sort of hardware-software codesign. We don't see transformer models as being an impediment in terms of memory requirements or speed in the near future.
[I would be happy to say more but I just got a top-level comment flag killed, presumably because they though I was advertising, so I won't mention the company name.]
If I understand correctly, Groq chips have 220MB SRAM and the next best level is DDR4? How many chips are needed to run Llama2-70B at those speeds?
Cool that you know the tech specs of the GroqChip! Yes, that's right, 220 MB of SRAM per chip. I think the demo where we first broke 200 tokens / sec was running on 1 GroqRack, so 64 chips. The live public demo that's currently running at 275 tokens / sec I think might be running on two GroqRacks, so 128 chips. I'm not certain of either of these figures so please don't quote me! But those are the right ball-park.
This article from less than a month ago says that it is on 576 chips https://www.nextplatform.com/2023/11/27/groq-says-it-can-dep...
Thanks, looks like you're right and this demo is running on 9 GroqRacks (576 chips). I think we may also have an 8 rack version in progress. We've tried a variety of different configurations to improve performance, which is possible because of the high level of flexibility and configurability of our architecture and compiler tool chain.
Are you guys hiring? I'm actually looking for a job in this industry.
Yes we are! (I put links in my profile because I'm worried if link here or mention the name I'll be flag killed again.)
Awesome, thanks! Is there someone there you could connect me with for an introductory chat? I put my contact info in my profile, I completely understand if it's not possible. Thanks!
Yes, sure! I've emailed a recruiter to introduce you. Thanks for reaching out and showing interest.
Link to HN discussion about the appliance: https://news.ycombinator.com/item?id=38739199
Re:How to make LLMs go fast
by pavelstoev on Friday December 22, 2023 @05:04PM
(#38739209)
There is still so much performance to gain by applying software-based optimizations for superior performance of transformer LLMs on GPUs. From ~ 30% GPU utilization to over 90% is possible. I wrote a white paper on this, page 5 and 6: http://tinyurl.com/uujr3z4f