Re:Self-Hosting Open Source LLMs: Cross Devices and Local De...
by 3Sophons on Tuesday November 21, 2023 @09:34PM
(#38373931)
The demos are as fast as the ChatGPT, they just look slower. -First. It might seem slow at first because it's loading the model. But once that's done, it gets much quicker for any follow-up requests. Second, Turning on streaming output helps a lot, as it shows responses as they're processed. And yes, the hardware matters too. A good GPU setup in the cloud can work wonders, though it might bump up the cost a bit. So actually the demo's not slower than ChatGPT overall; it just takes a moment to warm up at the start.
[ Parent
| on HN ]
There are no comments on this comment yet.