News for hackers. Stuff that matters.

A comment on Wikidata, with 12B facts, can ground LLMs to improve their factuality. Parent comment

Re:Wikidata, with 12B facts, can ground LLMs to improve thei... by unshavedyak on Friday November 17, 2023 @11:21AM (#38305643)
Yup. This is my gut to where LLMs will really explode. Let them augment data just a bit, train on improved data, augment more, train again - etcetc. If we take things slow i suspect in the long run it'll be really beneficial to multiple paradigms.

I know people say training bots on bot data is bad, but A: it's happening anyway, and B: it can't be worse than the actual garbage they get trained on in a lot of cases anyway.. can it?

[ Parent | on HN ]
Re:Wikidata, with 12B facts, can ground LLMs to improve thei... by gaogao on Friday November 17, 2023 @11:32AM (#38305831)
Training on bot data can be bad when it's ungrounded and basically hallucinations on hallucinations.

Having LLMs help curate something grounded is generally reasonable. Functionally, it's somewhat similar to how some training is using generated subtitles of videos for training video/text pairs; it's very feasible to also go and clean those up, even though it is bot data.

[ Parent | on HN ]
Re:Wikidata, with 12B facts, can ground LLMs to improve thei... by Filligree on Friday November 17, 2023 @11:29AM (#38305775)
Pixart-alpha provides an example of C: The bot labels can be dramatically better than the human labels.

Even though they used LLaVA, and LLaVA isn't all that good compared to gpt-4.

[ Parent | on HN ]
Re:Wikidata, with 12B facts, can ground LLMs to improve thei... by ori_b on Friday November 17, 2023 @02:43PM (#38309036)
> Yup. This is my gut to where LLMs will really explode

Yes, indeed. This is one place where LLMs can make it look like a bomb went off.

[ Parent | on HN ]
Re:Wikidata, with 12B facts, can ground LLMs to improve thei... by prosqlinjector on Friday November 17, 2023 @02:59PM (#38309155)
We can do polynomial regression of data sets that looks equally plausible, but it's not real data.
[ Parent | on HN ]
Re:Wikidata, with 12B facts, can ground LLMs to improve thei... by foobarchu on Friday November 17, 2023 @03:44PM (#38309978)
> A: it's happening anyway

This is never a valid defense for doing more of something.

[ Parent | on HN ]