News for hackers. Stuff that matters.
Re:You can now train a 70B language model at home by staticman2 on Sunday March 10, 2024 @10:52PM (#39664516)
A lot of people are reporting low context sizes tokens per second without discussing how slow it is at bigger sizes. In some cases they are also not mentioning time to first inference. If you dig around /LocalLLaMA you can find some posts with better bench-marking.

There are no comments on this comment yet.