GPT-3 has no idea what it’s talking about

Re:GPT-3 has no idea what it’s talking about by mordymoop on Saturday August 22, 2020 @01:43PM (#24245757)

GPT-3 involves zero fine-tuning or customization to any purpose. It’s not attempting to be a product, but rather a platform people can use to explore the possibilities of products. With fine-tuning it could do all sorts of domain-specific things that it can only passably do now. We suspect this to be true because we’ve seen how other systems behave with and without fine-tuning.

[ Parent | on HN ]

Re:GPT-3 has no idea what it’s talking about by ClumsyPilot on Saturday August 22, 2020 @05:01PM (#24247224)

you are sidestepping the question above - what can it be used for? What kind of fine-tuning is required? Because to me it looks like 'fixing catastrophic errors' not fine-tuning - as anywhere where the text can deliver value, it has to be correct.

AWS/Azure are a platform, and they work reliably. What kind of platform 'sometimes works'?

[ Parent | on HN ]

Re:GPT-3 has no idea what it’s talking about by voces on Saturday August 22, 2020 @05:44PM (#24247510)

Language models can help the disabled communicate better and faster (fine-tune on their online output, then the BCI can offer better continuations, so the patient does not have to type as much).

Fine-tune on educational materials, and a language model could be a 24/7 assistant to students.

Fine-tune on psychology or medicine data, and a language model could tell you of all medicine interactions, or act as a better ELIZA, and "socialize with" - and "support" people with depression or trauma.

Fine-tune on etiquette and social norms, and autistic people could ask questions without being ashamed.

These are some positive use cases. There are also neutral use cases (seeding a new social platform with autogenerated comments), and negative use cases (SEO spam, fake reviews, scaling up disinformation campaigns).

Edit: not complaining, but no idea why this was downvoted. Would be helpful to state why, so I don't make this mistake in the future (I am optimizing for useful replies).

[ Parent | on HN ]

Re:GPT-3 has no idea what it’s talking about by ClumsyPilot on Saturday August 22, 2020 @09:54PM (#24248907)

I did not downvote, but I think it's because you kinda missed the crux of my question is - what is fine-tuning, like what do you have to actually do?

Has anyone done it and demonstrated that it can be done on GPT to get useful and reliable output for some domain? If we exclude fake news/ twitter-bots or something equally silly.

[ Parent | on HN ]

Re:GPT-3 has no idea what it’s talking about by voces on Sunday August 23, 2020 @06:46PM (#24255933)

> what is fine-tuning, like what do you have to actually do?

For GPT-2 it is as easy as creating a corpus you want to fine-tune on, adding new unseen tokens to the dictionary, and run a finetuning command. https://huggingface.co/transformers/v1.2.0/examples.html#fin...

> Has anyone done it

Many have. AI Dungeon is a great example. Others are experimenting with humor generation, short story synopsis generation/creative writing, and supportive chat bots.

[ Parent | on HN ]