If I understand correctly, Groq chips have 220MB SRAM and the next best level is DDR4? How many chips are needed to run Llama2-70B at those speeds?

Cool that you know the tech specs of the GroqChip! Yes, that's right, 220 MB of SRAM per chip. I think the demo where we first broke 200 tokens / sec was running on 1 GroqRack, so 64 chips. The live public demo that's currently running at 275 tokens / sec I think might be running on two GroqRacks, so 128 chips. I'm not certain of either of these figures so please don't quote me! But those are the right ball-park.

This article from less than a month ago says that it is on 576 chips https://www.nextplatform.com/2023/11/27/groq-says-it-can-dep...

Thanks, looks like you're right and this demo is running on 9 GroqRacks (576 chips). I think we may also have an 8 rack version in progress. We've tried a variety of different configurations to improve performance, which is possible because of the high level of flexibility and configurability of our architecture and compiler tool chain.

Are you guys hiring? I'm actually looking for a job in this industry.

Yes we are! (I put links in my profile because I'm worried if link here or mention the name I'll be flag killed again.)

Awesome, thanks! Is there someone there you could connect me with for an introductory chat? I put my contact info in my profile, I completely understand if it's not possible. Thanks!

Yes, sure! I've emailed a recruiter to introduce you. Thanks for reaching out and showing interest.

Link to HN discussion about the appliance: https://news.ycombinator.com/item?id=38739199