OpenAI has made 1 major contribution - the auto regressive decoder
I've already introduced methods of instruct formatting within our org, because as the architectures change, what we learn from performing the work to organize and account for our internal knowledge will continuously pay off.
OpenAI is ahead on the RL data side. To me that’s the likely biggest advantage they have in GPT-4.
Palm-2 was never intended to match 4 (Gemini is). Llama 2 was never intended to match 4 either. Same with Claude 2. Nobody has matched GPT-4's training compute and failed to reach it.
> Diederik P. Kingma* University of Amsterdam, OpenAI [email protected]
They're bad at turning the research into a product.
Specifically, most AI based products need to be trained on user data, and Google is too scared to make any real use of much user data in AI models for fear of getting sued into oblivion if, for example, gmail can be tricked into writing something in an email which reveals some other gmail users private data.
Tbh, Google Lens is one of the best computer vision products in the market. Generally speaking, it's hard to crate a product around something that is completely new.
Transformers were invented in the early 90s: https://people.idsia.ch/~juergen/fast-weight-programmer-1991... Google were just the first with the compute to scale them up.
What models do they have to show after 15 years of AI research? AlphaGo, AlphaFold, and that's about it. Search is mediocre, translation the same, not even their OCR APIs are very good.
On pure research they invented the transformer in 2017 (Vaswani) and word embeddings in 2012 (Mikolov). But Microsoft invented residual connections (Kaiming He 2015) which underline transformers and modern CNNs. CNNs and LSTMs were invented longe before, by Yann LeCun (1989) and Jürgen Schmidhuber (1997).
And yes, OpenAI also invented something of their own - the Adam optimiser (Kingma 2015), that trains 99% of the models today.
Hinton invented Dropout (2014) and backpropagation (1986).