News for hackers. Stuff that matters.

A comment on Llamafile lets you distribute and run LLMs with a single file. Parent comment

Re:Llamafile lets you distribute and run LLMs with a single ... by brucethemoose2 on Wednesday November 29, 2023 @06:01PM (#38466553)
Basically enough to fit the download in RAM + a bit more.

In practice, you kinda need a GPU, even a small one. Otherwise prompt processing is really slow.

[ Parent | on HN ]
Re:Llamafile lets you distribute and run LLMs with a single ... by wazoox on Sunday December 03, 2023 @09:34AM (#38507343)
It's really decent without any GPU. Image analysis is somewhat long, but text prompts are fine. My Ryzen laptop does 2.5 to 4 tokens per second, my Mac pro more like 8.
[ Parent | on HN ]