News for hackers. Stuff that matters.

A comment on OpenAI’s policies hinder reproducible research on language models. Parent comment

Re:OpenAI’s policies hinder reproducible research on languag... by Tenoke on Thursday March 23, 2023 @04:26AM (#35271809)
The quality with reformer is much much worse, it's not really comparable.
[ Parent | on HN ]
Re:OpenAI’s policies hinder reproducible research on languag... by bitL on Thursday March 23, 2023 @05:27AM (#35272151)
Yeah, but it fits on a single GPU. Now imagine it scaled across 1000 GPUs.
[ Parent | on HN ]
Re:OpenAI’s policies hinder reproducible research on languag... by Tenoke on Thursday March 23, 2023 @06:09AM (#35272407)
I finetuned one in 2020[0] to play around with it and the results still seemed a bit worse than a gpt of comparable size.

0. https://svilentodorov.xyz/blog/reformer-99m/

[ Parent | on HN ]