News for hackers. Stuff that matters.
Re:Show HN: I've built a locally running Perplexity clone by nilsherzig on Thursday April 04, 2024 @04:41AM (#39927915)
You could run the LLM using your CPU and normal (non video) ram. But that's a lot slower. There are people working on making it a lot faster tho. The bottleneck is the transfer speed between the ram Sticks and the CPU.

Just taking a guess, but I wouldn't expect more than a couple tokens (more or less like syllables) per second. Which is probably to slow, since it has to read a couple thousand per search result.

It's hard to provide minimum requirements, since there are so many edge cases.

There are no comments on this comment yet.