News for hackers. Stuff that matters.
Re:Localllm lets you develop gen AI apps on local CPUs by 3abiton on Thursday February 08, 2024 @03:51AM (#39299795)
What's the speed? I heard of llamafile before.
Re:Localllm lets you develop gen AI apps on local CPUs by depingus on Thursday February 08, 2024 @11:07AM (#39303600)
I wasn't really timing it, but it felt fine, especially since it prints out while generating. The 13B model on the 5600X is about 10 seconds. Tho once the conversation gets too long (about 10 replies) it will go sideways. I gotta play with the default settings. IIRC the 7B on the Fx8350 was closer to 25 seconds per response.