News for hackers. Stuff that matters.

A comment on OpenAI’s policies hinder reproducible research on language models. Parent comment

Re:OpenAI’s policies hinder reproducible research on languag... by redox99 on Wednesday March 22, 2023 @11:13PM (#35270112)
Don't forget about a more efficient attention that let's them get 32k tokens of context.
[ Parent | on HN ]
Re:OpenAI’s policies hinder reproducible research on languag... by bitL on Wednesday March 22, 2023 @11:45PM (#35270308)
It's still much worse than 1M context on 16GB VRAM with Reformer, but at the cost of inference speed. And you can use FlashAttention in your own models to get a more efficient/sparse attention now as well.
[ Parent | on HN ]
Re:OpenAI’s policies hinder reproducible research on languag... by meghan_rain on Thursday March 23, 2023 @01:12AM (#35270753)
How could one apply the mentioned technologies to llama/alpaca?
[ Parent | on HN ]
Re:OpenAI’s policies hinder reproducible research on languag... by Tenoke on Thursday March 23, 2023 @04:26AM (#35271809)
The quality with reformer is much much worse, it's not really comparable.
[ Parent | on HN ]
Re:OpenAI’s policies hinder reproducible research on languag... by bitL on Thursday March 23, 2023 @05:27AM (#35272151)
Yeah, but it fits on a single GPU. Now imagine it scaled across 1000 GPUs.
[ Parent | on HN ]
Re:OpenAI’s policies hinder reproducible research on languag... by Tenoke on Thursday March 23, 2023 @06:09AM (#35272407)
I finetuned one in 2020[0] to play around with it and the results still seemed a bit worse than a gpt of comparable size.

0. https://svilentodorov.xyz/blog/reformer-99m/

[ Parent | on HN ]