News for hackers. Stuff that matters.

A comment on You can now train a 70B language model at home. Parent comment

Re:You can now train a 70B language model at home by int_19h on Friday March 08, 2024 @02:41PM (#39645360)
"Real-time" is a very vague descriptor. I get 7-8 tok/s for 70b model inference on my M1 Mac - that's pretty real-time to me. Even Professor-155b runs "good enough" (~3 tok/s) for what I'd consider real-time chat in English.
[ Parent | on HN ]

There are no comments on this comment yet.