News for hackers. Stuff that matters.
Re:A comprehensive guide to running Llama 2 locally by zacmps on Tuesday July 25, 2023 @08:49PM (#36871758)
70B at f16 needs about 120-160GB (4 A100 40GB or 2 A100 80GB).

Quantization still seems to have some issues for the 70B model.

There are no comments on this comment yet.