Re:GPT-Neo – Building a GPT-3-sized model, open source and free
by qayxc on Monday January 18, 2021 @07:20AM
(#25821015)
This. Plus the increasing amount of intransparent results.
Training data is private, so it's impossible to even try to recreate results, validate methods, or find biases/failure cases.
[ Parent
| on HN ]
Re:GPT-Neo – Building a GPT-3-sized model, open source and free
by stellaathena on Tuesday January 19, 2021 @08:29AM
(#25833217)
This is dead right and very important. The data you train on is much more important than model architecture in terms of validation and compliance, and yet it’s a closely held secret. Producing or obtaining good data is a pain in the ass.
For this reason, EAI has made the data we are training on public. I can’t link to it because of anon policies at conferences, but if you look at our website I’m sure you can find a paper detailing it and a link to download it.