OpenAI’s policies hinder reproducible research on language models

Re:OpenAI’s policies hinder reproducible research on languag... by ShamelessC on Wednesday March 22, 2023 @10:42PM (#35269883)

Yannic Kilcher's opinion on this is likely correct. Similar parameter count, but trained for longer. The particulars of their instruction tuning/whatever-else-they-did are the real secret sauce.

[ Parent | on HN ]

Re:OpenAI’s policies hinder reproducible research on languag... by redox99 on Wednesday March 22, 2023 @11:13PM (#35270112)

Don't forget about a more efficient attention that let's them get 32k tokens of context.

[ Parent | on HN ]

Re:OpenAI’s policies hinder reproducible research on languag... by bitL on Wednesday March 22, 2023 @11:45PM (#35270308)

It's still much worse than 1M context on 16GB VRAM with Reformer, but at the cost of inference speed. And you can use FlashAttention in your own models to get a more efficient/sparse attention now as well.

[ Parent | on HN ]

Re:OpenAI’s policies hinder reproducible research on languag... by meghan_rain on Thursday March 23, 2023 @01:12AM (#35270753)

How could one apply the mentioned technologies to llama/alpaca?

[ Parent | on HN ]

Re:OpenAI’s policies hinder reproducible research on languag... by Tenoke on Thursday March 23, 2023 @04:26AM (#35271809)

The quality with reformer is much much worse, it's not really comparable.

[ Parent | on HN ]

Re:OpenAI’s policies hinder reproducible research on languag... by bitL on Thursday March 23, 2023 @05:27AM (#35272151)

Yeah, but it fits on a single GPU. Now imagine it scaled across 1000 GPUs.

[ Parent | on HN ]

Re:OpenAI’s policies hinder reproducible research on languag... by Tenoke on Thursday March 23, 2023 @06:09AM (#35272407)

I finetuned one in 2020[0] to play around with it and the results still seemed a bit worse than a gpt of comparable size.

0. https://svilentodorov.xyz/blog/reformer-99m/

[ Parent | on HN ]