You can now train a 70B language model at home

Re:You can now train a 70B language model at home by AnthonyMouse on Friday March 08, 2024 @04:02PM (#39646444)

> It's fast enough to do realtime (7 tok/s) chat with 120b models.

Please list quantization for benchmarks. I'm assuming that's not the full model because that would need 256GB and I don't see a Studio model with that much memory, but q8 doubles performance and q4 quadruples it (with corresponding loss of quality).

> But in practice, I'm not aware of any actual offerings in this space for comparable money that do not involve having to mess around with building things yourself.

You can just buy a complete server from a vendor or eBay, but this costs more because they'll try to constrain you to a particular configuration that includes things you don't need, or overcharge for RAM etc. Which is basically the same thing Apple does.

Whereas you can buy the barebones machine and then put components in it, which takes like fifteen minutes but can save you a thousand bucks.

[ Parent | on HN ]

Re:You can now train a 70B language model at home by int_19h on Friday March 08, 2024 @06:27PM (#39647774)

That is with 4-bit quantization. For practical purposes I don't see the point of running anything higher than that for inference.

[ Parent | on HN ]

Re:You can now train a 70B language model at home by AnthonyMouse on Friday March 08, 2024 @07:19PM (#39648121)

That's interesting though, because it implies the machine is compute-bound. A 4-bit 120B model is ~60GB, so you should get ~13 tokens/second out of 800GB/s if it was memory-bound. 7/s implies you're getting ~420GB/s.

And the Max has half as many cores as the Ultra, implying it would be compute-bound too.

[ Parent | on HN ]

Re:You can now train a 70B language model at home by sbierwagen on Saturday March 09, 2024 @12:54AM (#39649718)

6.3 tok/s has been demonstrated on q4_0 Falcon 180B on the 192gb Mac studio: https://x.com/ggerganov/status/1699791226780975439?s=46&t=ru...

[ Parent | on HN ]

Re:You can now train a 70B language model at home by staticman2 on Sunday March 10, 2024 @10:52PM (#39664516)

A lot of people are reporting low context sizes tokens per second without discussing how slow it is at bigger sizes. In some cases they are also not mentioning time to first inference. If you dig around /LocalLLaMA you can find some posts with better bench-marking.

[ Parent | on HN ]