News for hackers. Stuff that matters.
Re:LLaMA Now Goes Faster on CPUs by vlunkr on Monday April 01, 2024 @01:18AM (#39891129)
We’re not talking about running chatGPT locally though, are we?
Re:LLaMA Now Goes Faster on CPUs by fragmede on Monday April 01, 2024 @01:22AM (#39891147)
sigh your going to make me open my laptop, aren't you.
Re:LLaMA Now Goes Faster on CPUs by fragmede on Monday April 01, 2024 @01:36AM (#39891197)
I ran 'who won the 1986 Cricket World Cup' against llama2-uncensored (the local model I have pre-downloaded) and hilarious got 5 different answers asking it 5 times:

    >>> who won the 1986 Cricket World Cup
    India
    
    >>> who won the 1986 Cricket World Cup
    Australia
    
    >>> who won the 1986 Cricket World Cup
    New Zealand
    
    >>> who won the 1986 Cricket World Cup
    West Indies
    
    >>> who won the 1986 Cricket World Cup
    England
Which proves GP's point about hallucinations, though none of those are

> Brooklyn Nets won the 1986 Cricket World Cup by scoring 46 yards in only 3 frames

LLM's hallucinations are insidous because they have the ring of truth around them. yards and frames aren't cricket terms, so we're off to the races with them.

Re:LLaMA Now Goes Faster on CPUs by astrange on Monday April 01, 2024 @02:24AM (#39891365)
If you want factual answers from a local model it might help to turn the temperature down.
Re:LLaMA Now Goes Faster on CPUs by fragmede on Monday April 01, 2024 @04:05AM (#39891792)
It would also help if I had more VRAM and wasn't running a 7B parameter 4-bit quantized model.
Re:LLaMA Now Goes Faster on CPUs by jrflowers on Monday April 01, 2024 @02:43AM (#39891455)
> If you want factual answers from a local model it might help to turn the temperature down.

This makes sense. If you interact with a language model and it says something wrong it is your fault

Re:LLaMA Now Goes Faster on CPUs by astrange on Monday April 01, 2024 @04:52AM (#39892011)
You're not "interacting with a language model", you're running a program (llama.cpp) with a sampling algorithm which is not set to maximum factualness by default.

It's like how you have to set x264 to the anime tuning or the film tuning depending on what you run it on.

Re:LLaMA Now Goes Faster on CPUs by beefnugs on Monday April 01, 2024 @03:13AM (#39891587)
Actually isn't this good? It means we can run something multiple times to prove itself a bad answer?
Re:LLaMA Now Goes Faster on CPUs by latexr on Monday April 01, 2024 @08:15AM (#39893299)
You can ask LLMs the same question and they might sometimes get it wrong and other times get it right. Having different answers is no indication that none of them is correct.

Furthermore, even if an LLM always gives the same answer to a question, there’s no guarantee the answer is correct.

https://en.wikipedia.org/wiki/Propaganda

https://en.wikipedia.org/wiki/Big_lie#Alleged_quotation

Re:LLaMA Now Goes Faster on CPUs by sroussey on Monday April 01, 2024 @01:45PM (#39896847)
An LLM will always give the same output for the same input. It’s sorta like a random number generator that gives the same list of “random” numbers for the same seed. LLMs get a seed too.
Re:LLaMA Now Goes Faster on CPUs by latexr on Monday April 01, 2024 @04:46PM (#39899109)
That’s irrelevant for the matter. The person I replied to obviously did not have seeded responses in mind.
Re:LLaMA Now Goes Faster on CPUs by saagarjha on Tuesday April 02, 2024 @11:56PM (#39913465)
It can tell you if the answer is wrong, but it can never tell you if the answer is right.
Re:LLaMA Now Goes Faster on CPUs by ilaksh on Monday April 01, 2024 @09:26AM (#39893952)
You should specify the model size and temperature.

For fact retrieval you need to use temperature 0.

If you don't get the right facts then try 34b, 70b, Mixtral, Falcon 180b, or another highly ranked one that has come out recently like DBRX.