Re:You can now train a 70B language model at home
by int_19h on Friday March 08, 2024 @06:27PM
(#39647774)
That is with 4-bit quantization. For practical purposes I don't see the point of running anything higher than that for inference.
[ Parent
| on HN ]
Re:You can now train a 70B language model at home
by AnthonyMouse on Friday March 08, 2024 @07:19PM
(#39648121)
That's interesting though, because it implies the machine is compute-bound. A 4-bit 120B model is ~60GB, so you should get ~13 tokens/second out of 800GB/s if it was memory-bound. 7/s implies you're getting ~420GB/s.
And the Max has half as many cores as the Ultra, implying it would be compute-bound too.