News for hackers. Stuff that matters.
Re:GPU-Accelerated LLM on a $100 Orange Pi by brucethemoose2 on Tuesday August 15, 2023 @10:22AM (#37134354)
If you can run it as a AI Horde worker, and the home usage is sporadic, you could definitely support more than one person.

Otherwise ~1.5 tokens/s is definitely the minimum you'd want streaming tokens to a single person.

There are no comments on this comment yet.