News for hackers. Stuff that matters.

A comment on GPT Neo: open-source GPT model, with pretrained 1.3B & 2.7B weight models. Parent comment

Re:GPT Neo: open-source GPT model, with pretrained 1.3B & 2.... by Voloskaya on Sunday March 21, 2021 @07:27PM (#26535096)
It's trained using the same architecture, and with a very similar dataset, so it should be very close.
[ Parent | on HN ]
Re:GPT Neo: open-source GPT model, with pretrained 1.3B & 2.... by dylanbyte on Sunday March 21, 2021 @10:31PM (#26536248)
My experience is that replicating papers is actually nontrivial. For example someone announced they had replicated gpt2 some time back but when evals were run it turned about to be the equivalent of a much smaller model.
[ Parent | on HN ]