News for hackers. Stuff that matters.
Re:The GPT Architecture, on a Napkin by sebzim4500 on Sunday December 11, 2022 @11:19AM (#33944486)
It feels like this is less true for GPT though, especially as OpenAI seems to be adopting a 'kitchen sink' approach.
Re:The GPT Architecture, on a Napkin by mike_hearn on Sunday December 11, 2022 @01:54PM (#33946177)
Just getting plain text out of the web without getting flooded with boilerplate, noise, SEO spam, duplication, infinity pages like calendars etc is already a hard data engineering problem.
Re:The GPT Architecture, on a Napkin by visarga on Sunday December 11, 2022 @06:46PM (#33948951)
You're only thinking of the training data. But the pre-trained model is like a newborn, trashing and yelling and not listening. It needs a second level of training made of a mix of about 1800 supervised tasks. Now it has progressed a little, you can get it to listen, but it's still not ok, it's like a 5 year old. You need to label more data with human preferences and fine-tune the model to align it with what we think is good behaviour. Now it behaves like a 10 year old.

In the original dataset you already combine dozens of sources - web scrapes, book collections, paper collections, materials in many languages, etc. In the second stage you have thousands of small supervised datasets. In the third stage you have to label. So I think the dataset building phase is pretty difficult.