News for hackers. Stuff that matters.

A comment on Run the strongest open-source LLM model: Llama3 70B with just a single 4GB GPU. Parent comment

Re:Run the strongest open-source LLM model: Llama3 70B with ... by nutrientharvest on Wednesday June 26, 2024 @02:41PM (#40803122)
Ollama can already run Llama-3 70B with a 4GB GPU, or no GPU at all, it'll just be slow.

Considering this says it's "not designed for real-time interactive scenarios" it's probably also really slow

[ Parent | on HN ]
Re:Run the strongest open-source LLM model: Llama3 70B with ... by cpill on Wednesday June 26, 2024 @03:18PM (#40803499)
so how much GPU RAM does need to get the 70B going fast (ish)?
[ Parent | on HN ]
Re:Run the strongest open-source LLM model: Llama3 70B with ... by AaronFriel on Wednesday June 26, 2024 @03:22PM (#40803544)
A good rule of thumb is that models can be quantized to 6 to 8 bits per weight without significantly degrading quality. This is convenient for the math: 70GB plus some overhead for the attention matrices (ongoing requests). This overhead depends on workload and context lengths, but you should expect about 30% more. So, around 100GB for a server under load.
[ Parent | on HN ]