Are you just talking about transformers? OpenAI utilized them, yes, but they are the ones who authored the original GPT paper. I am not sure I follow.
Re:Planning for AGI and Beyond
by lobstersammich on Friday February 24, 2023 @05:25PM
(#34931011)
I think they mean Transformers in the Vaswani et al 'Attention is all you need' paper, not Generative Pretrained Transformers, specifically? Paper link below:
https://proceedings.neurips.cc/paper/2017/file/3f5ee243547de...
For some papers on attention mechanisms from before the 2017 'Attention is all you need' paper, check out that paper's references. Chris Manning's 2015 paper covers attention mechanisms. And so do a few other researchers from that mid-2010s time period:
[21] Minh-Thang Luong, Hieu Pham, and Christopher D Manning. Effective approaches to attention based neural machine translation. arXiv preprint arXiv:1508.04025, 2015.
[22] Ankur Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. A decomposable attention model. In Empirical Methods in Natural Language Processing, 2016.
My understanding is that the original Transformer paper ("Attention is All You Need") and BERT papers came from Google. In particular, BERT's pre-training task is to predict how to fill-in-the-blank from a sentence that's missing a random word. OpenAI started from that architecture and made it output text generatively, by using beam search on top of a next-token prediction task.
It attracts more investment
I share the same opinion. While ChatGPT might be a great showcase of LLM, its greatest achievement could be its ability to draw VCs attention and money to whatever idea as long as you stick a ".ai" domain next to it.
What's more baffling to me is that quite a few software engineers (or at least few that I know and appreciate) think that generative AI somehow suggest that AGI might be just around the corner or that it'll somehow be but an iteration over existing models when I fail to see why we should assume these two problems are of a similar order of magnitude.
Who do these people think they are? They didn’t even invent the architecture GPT3 is built on! This is as bad or worse than Elon claiming full self driving is just around the corner for almost a decade!