News for hackers. Stuff that matters.

A comment on Llama 3.1 Official Launch. Parent comment

Re:Llama 3.1 Official Launch by lolinder on Tuesday July 23, 2024 @10:58AM (#41046647)
Quantized to 4 bits you'll only need ~200GB! 5 4090s should cover it.
[ Parent | on HN ]
Re:Llama 3.1 Official Launch by woodson on Tuesday July 23, 2024 @11:04AM (#41046726)
I wonder if AutoAWQ works out of the box, given no architectural changes (?). That would be most straightforward together with vLLM for serving.
[ Parent | on HN ]
Re:Llama 3.1 Official Launch by pat2man on Tuesday July 23, 2024 @11:29AM (#41047028)
Two 128gb Mac studios networked via thunderbolt 4?
[ Parent | on HN ]
Re:Llama 3.1 Official Launch by Teknomancer on Tuesday July 23, 2024 @11:45AM (#41047267)
This is actually a promising endeavor. Id love to see someone try that.
[ Parent | on HN ]
Re:Llama 3.1 Official Launch by angoragoats on Tuesday July 23, 2024 @12:00PM (#41047485)
There's already at least one project that attempts this:

https://github.com/exo-explore/exo

[ Parent | on HN ]
Re:Llama 3.1 Official Launch by angoragoats on Tuesday July 23, 2024 @10:59AM (#41046665)
You'll probably need 9 or more. 4090s have 24GB each.
[ Parent | on HN ]
Re:Llama 3.1 Official Launch by lolinder on Tuesday July 23, 2024 @11:12AM (#41046817)
Oops, I read 48 somewhere but that's wrong. Thanks.
[ Parent | on HN ]
Re:Llama 3.1 Official Launch by htrp on Tuesday July 23, 2024 @11:29AM (#41047031)
a6ks however
[ Parent | on HN ]
Re:Llama 3.1 Official Launch by downvotetruth on Tuesday July 23, 2024 @01:53PM (#41048875)
If an implementation had NVidia's Heterogeneous Memory Management implemented, then 192 GB RAM DDR5 + GPU VRAM would seem to be close.
[ Parent | on HN ]