Re:Ask HN: Which LLMs can run locally on most consumer compu...
by imtringued on Thursday May 23, 2024 @02:34AM
(#40451211)
There isn't. For games you would need vLLM, because batch size is more important than latency. Something that people don't seem to understand is that an NPC doesn't need to generate tokens faster than its TTS can speak. You only need to minimize the time to first token.
[ Parent
| on HN ]
There are no comments on this comment yet.