News for hackers. Stuff that matters.
Re:You can now train a 70B language model at home by sbierwagen on Saturday March 09, 2024 @12:54AM (#39649718)
6.3 tok/s has been demonstrated on q4_0 Falcon 180B on the 192gb Mac studio: https://x.com/ggerganov/status/1699791226780975439?s=46&t=ru...
Re:You can now train a 70B language model at home by staticman2 on Sunday March 10, 2024 @10:52PM (#39664516)
A lot of people are reporting low context sizes tokens per second without discussing how slow it is at bigger sizes. In some cases they are also not mentioning time to first inference. If you dig around /LocalLLaMA you can find some posts with better bench-marking.