News for hackers. Stuff that matters.

A comment on Running large language models like ChatGPT on a single GPU. Parent comment

Re:Running large language models like ChatGPT on a single GPU by borzunov on Monday February 20, 2023 @08:04PM (#34875143)
I'm afraid that, unlike proprietary APIs and Petals, this system can't be used for single-batch inference of 175B models with interactive speeds - the thing you actually need for running ChatGPT and other interactive LM apps. See https://news.ycombinator.com/item?id=34874976
[ Parent | on HN ]

There are no comments on this comment yet.