News for hackers. Stuff that matters.
Re:LLaMA Now Goes Faster on CPUs by devsda on Sunday March 31, 2024 @11:54PM (#39890763)
For someone interested in learning about LLMs, running them locally is a good way to understand the internals.

For everyone else, I wish they experience these (locally or elsewhere) weak LLMs atleast once before using the commercial ones just to understand various failure modes and to introduce a healthy dose of skepticism towards the results instead of blindly trusting them to be the facts/truth.

Re:LLaMA Now Goes Faster on CPUs by simonw on Sunday March 31, 2024 @11:55PM (#39890770)
Completely agree. Playing around with a weak LLM is a great way to give yourself a little bit of extra healthy skepticism for when you work with the strong ones.
Re:LLaMA Now Goes Faster on CPUs by mmahemoff on Monday April 01, 2024 @01:59AM (#39891274)
How do you learn about the internals by running LLMs locally? Are you playing with The code, runtime params, or just interacting via chat?
Re:LLaMA Now Goes Faster on CPUs by samus on Monday April 01, 2024 @02:24AM (#39891364)
The abstractions are relatively brittle. If you don't have a powerful GPU, you will be forced to consider how to split the model between CPU and GPU, how much context size you need, whether to quantize the model, and the tradeoffs implied by these things. To understand these, you have to develop a basic model how an LLM works.
Re:LLaMA Now Goes Faster on CPUs by barrkel on Monday April 01, 2024 @06:41AM (#39892630)
By interacting with it. You see the contours of its capabilities much more clearly, learn to recognize failure modes, understand how prior conversation can set the course of future conversation in a way that's almost impossible to correct without starting over or editing the conversation history.
Re:LLaMA Now Goes Faster on CPUs by samus on Monday April 01, 2024 @02:20AM (#39891348)
This skepticism is completely justified since ChatGPT 3.5 is also happily hallucinating things that don't exist. For example how to integrate a different system Python interpreter into pyenv. Though maybe ChatGPT 4 doesn't :)