News for hackers. Stuff that matters.
Re:Training and Aligning LLMs with RLHF and RLHF Alternatives by jamesblonde on Sunday September 10, 2023 @02:51PM (#37458784)
I read here that Yann LeCun claimed that even with RLHF, LLMs will still hallucinate - that it's an unavoidable consequence of their autoregressive nature

https://www.hopsworks.ai/dictionary/rlhf-reinforcement-learn...

Re:Training and Aligning LLMs with RLHF and RLHF Alternatives by og_kalu on Sunday September 10, 2023 @03:12PM (#37458973)
Likely yes. But "solving" hallucinations is not really important as long as mitigating it to some sufficiently low level is possible.
Re:Training and Aligning LLMs with RLHF and RLHF Alternatives by bugglebeetle on Sunday September 10, 2023 @04:01PM (#37459434)
For many NLP tasks (which is what I mostly use LLMs for), hallucinations can be prevented with simple, procedural checks against the input. For example, for NER tasks, you can just check whether the extracted entities are in the original input.
Re:Training and Aligning LLMs with RLHF and RLHF Alternatives by phillipcarter on Sunday September 10, 2023 @04:19PM (#37459628)
Moreover, it's all about use case. If you need a high degree of reliability and reproducibility, don't use LLMs! Not yet, at least. That's fine though, because there's a ton of value they offer in solving problems where that isn't needed.
Re:Training and Aligning LLMs with RLHF and RLHF Alternatives by bugglebeetle on Sunday September 10, 2023 @04:50PM (#37459944)
> If you need a high degree of reliability and reproducibility, don't use LLMs!

This is true of pretty much all of machine learning. LLMs are just getting singled out because their outputs are not getting the same level of validation that typicall occurs with older approaches. BERT models will also spit out whacky stuff, depending on how they’re trained/fine-tuned/used/etc

Re:Training and Aligning LLMs with RLHF and RLHF Alternatives by 3abiton on Sunday September 10, 2023 @05:04PM (#37460073)
I wonder if there will be a new metric implemented in evaluating LLMs: Hallucination score.
Re:Training and Aligning LLMs with RLHF and RLHF Alternatives by westurner on Monday September 11, 2023 @02:36AM (#37463686)
When the next token is a URL, and the URL does not match the preceding anchor text.

Additional layers of these 'LLMs' could read the responses and determine whether their premises are valid and their logic is sound as necessary to support the presented conclusion(s), and then just suggest a different citation URL for the preceding text.

"#StructuredPremises"

Re:Training and Aligning LLMs with RLHF and RLHF Alternatives by ShamelessC on Sunday September 10, 2023 @03:10PM (#37458947)
That goes without saying.

edit: I don't like your linked article at all. Subtly misleading and/or misinformed. Like a yahoo news but for ML.

to clarify: No one (certainly not OpenAI) ever suggested that RLHF was even remotely useful at dealing with the problem of hallucinations. It's not for that. The insinuation that it was designed for that purpose (at least partially) and yet _failed_ is a faulty one. It was not designed for that purpose. Hallucinations are a known issue with large language models, and while I appreciate LeCunn re-iterating that; far lesser researchers than LeCunn are aware of that fact.

Re:Training and Aligning LLMs with RLHF and RLHF Alternatives by jprete on Sunday September 10, 2023 @10:30PM (#37462170)
What does hallucination have to do with alignment?
Re:Training and Aligning LLMs with RLHF and RLHF Alternatives by ftxbro on Monday September 11, 2023 @12:28AM (#37462876)
what people in the media are calling 'hallucination' in large language models is really imagination and creativity. it's inherent in the models and it's why they have been able to unlock so many amazing cognitive capabilities where previous efforts have failed
Re:Training and Aligning LLMs with RLHF and RLHF Alternatives by Salgat on Monday September 11, 2023 @01:03AM (#37463087)
That's like saying that a simple linear regression predicting the wrong (x,y) value is just "being creative". The LLM is lacking in sufficient complexity to make the correct word prediction. It's not being creative, it is just flat out wrong.
Re:Training and Aligning LLMs with RLHF and RLHF Alternatives by ftxbro on Monday September 11, 2023 @09:28AM (#37466956)
Sure your linear regression example is great, and it's an example of how 'hallucination' in models can be good. Imagine if the only prediction algorithms had to go through every point in noisy data, it would look like either some overfitted thing like an ugly high order polynomial that goes through every point or just a database of every x y pair in the training set. Neither of those ones hallucinate if you ask the data from the training set. But for most purposes they wouldn't be as useful as a fitted linear regression, which 'hallucinates' y values even if you give it exact x from the training set. Of course if you just want to memorize and repeat back the training set then just a database is the best way!
Re:Training and Aligning LLMs with RLHF and RLHF Alternatives by Salgat on Monday September 11, 2023 @11:42AM (#37468959)
So by your definition of creativity, the LLM is always being creative because it's always using probability to determine the next word (it's not just spitting out facts from a database). Hallucination is just when that prediction, or creativity as you call it, is wrong.
Re:Training and Aligning LLMs with RLHF and RLHF Alternatives by scoresmoke on Sunday September 10, 2023 @05:55PM (#37460490)
Discussions about LLM alignment often miss topics of data quality and quantity. It turns out that current models like Llama 2 use 10K+ prompts and responses for supervised fine-tuning (SFT) and 100K+ human preference pairs. While the preferences are pretty easy to annotate, producing a good SFT dataset is uneasy.

https://evalovernite.substack.com/p/rlhf-math-aint-enough

https://doi.org/10.5281/zenodo.8186168

Re:Training and Aligning LLMs with RLHF and RLHF Alternatives by Geee on Sunday September 10, 2023 @06:42PM (#37460852)
What datasets OpenAI uses for RLHF? Is the assumption correct that it's "time & labor intensive"? Couldn't you take ranked responses from HN / Reddit / Stack Exchange / Quora etc. where answers are already ranked, and train the reward model on that?