Catching up right now is not a matter of tech innovation, but raw energy and compute.

Of course, the next -revolution- in AI could very well come from Ilya. But why would he bestow that honor to anyone? He can self fund it if he wants. It's an R&D project, not a scaling problem.

> Catching up right now is not a matter of tech innovation, but raw energy and compute.

I don't really understand that. ChatGPT is not all that impressive as an actual tool for many things. It doesn't really seem to matter to me what energy and compute is going into it. Will it make it actually work?

If you consider the best frontier LLM in the world "not that impressive," I'm curious to know what you think is

I think the point is not "chatGPT isn't good", it's that no one is beating physics, you can rough out the coefficients on a napkin, and they're playing chicken with who wants to set more piles of money on fire.

It _is_ the best frontier LLM in the world, and virtually the entire global population of people who care about that are in this thread

This will change when the rumors turn out to be true and Apples Siri is powered by OpenAI and a billion people have a working conversational AI in their pocket.

I agree that this will happen (I said in another post), but I can't for the life of me attach it to one of the points I made

If that is the case, why hasn't Google caught up yet?

Is Google significantly behind? I would say from a technological perspective, they're very closely trailing or even surpassing OpenAI in many ways; they've built a formidable ChatGPT competitor in Gemini—not to mention they have a huge home court advantage with billions of people on Search, Chrome, Android, and G Suite. If you zoom out a bit, it's not really a fair fight between the two. More likely, Apple and Microsoft win against Google because they use/buy OpenAI.

Technical chops has never been Google's problem. Gemini for all the hype Google has thrown at it, has continued the recurring trend of G not knowing how to win with their product launches.

When you push either model to its limits, yes, Google is pretty far behind GPT-4 with Gemini.

Because their weakness is in focus and execution, not resources.

If they focused all of their energy on, simply, a frontier AI model, and not trying to shoehorn a half-complete model into all of their products, there is no doubt they would be ahead.

But this is the innovator's dilemma, and why it is that startups are the disruptors.

Big companies move slow and lack focus. Small companies move fast and can only focus on one thing.

and politics, although they seem to have woken up (no pun intended) and started de-politicizing.

Because Google is huge. Presumably the same reason why Microsoft finds it easier to shovel billions into OpenAI instead of doing it in-house.

Wrong leadership at every level: Sundar as CEO, Prabhakar as Search SVP, Sissie as Assistant/Gemini VP. Maybe they should hire Ilya instead?

my guess is that a significantly underrated reason is that Google holds itself to stricter data compliance standards than OAI

That is openly laughable. Incognito mode claims, just one of a long list of examples of this politicized monopoly engaging in criminal behavior.

I don’t think that’s a fair take nor does it really disprove mine.

The compliance regime at big corps is definitely more sophisticated than at a player like OAI

Cynicism is just giving yourself an excuse to be wrong because it sounds cooler.

Google is ideologically compromised which ties them into knots and creates endless internal struggles.

this led to the release of Gemini which was absurdly biased when viewed by the typical American.

probably because they have identified (correctly) that slowly integrating AI into their existing products that make them hundreds of billions is smarter than just burning that money to get upvoted on HN. Tall trees catch too much wind, ancient Chinese proverb. If you can afford to being second is usually less painful than being first.

There are clear advantages and disadvantages to that approach, but assuming this is the correct choice ( seems plausible, but I am not automatically convinced ), is that the actual reason for being as behind as they are?

I would argue that is not the case. I won't re-list some of the reasons other posters mentioned, which, based on past year, appear more likely ( decisions hamstrung by corporate committees, data governance bureaucracy, and last, but not least, ideology focus ). Leadership that is actually focused on 'delivering value to the shareholder' or not being worried about first mover advantage seems only a part of it.

edit: added first mover wording

I feel like the lowest hanging fruit right now lies with the UI and already established techniques for reducing latency and making the experience smoother.

Just executing the client right would give someone a competitive advantage right now...

I'd argue it's mainly a matter of people. Otherwise Meta, Amazon, Google, etc would've released their versions like yesterday.

I mean, that's probably 90% or 95% true, but the remaining 5-10% is almost certainly worth a $100M offer to Sutskever from Google or Meta (or possibly Amazon or Apple).

it’s a matter of data

You will still need the compute resources of Microsoft, Meta etc.

And they have their own people who equally know how LLMs work.

Even raising funds is not a certainty given that VCs are becoming more cautious with AI as they realise it's now a platform fight between the mega corporations.

Plenty of companies have compute, but everyone is barely catching up to GPT-4 over a year after release. I'm sure places like Meta would love to unlock the details of what makes it so good.

There's no mystery, they just have a lot of actual user data, so they've been able to refine the question answering behavior. They've also baked some common problem solving strategies like chain of thought into the model via training.

yep, and the puzzle isnt 'make chatgtp again' its 'catch up to chatgpt' which is a whole other puzzle

> Even raising funds is not a certainty given that VCs are becoming more cautious

You'd think any investor would be utterly stupid to not make an exception for Ilya, regardless.

That’s not such a given. There are a toooon of deep pockets in this space already and absolutely zero path to profitable products for any of them.

If his concern was irresponsible AI proliferation, too much commercial focus and wanting to move more carefully, accelerating competition doesn't seem like it would align with his goals.

If he really is quitting over ethics he isn't going to Google or Meta. He'll probably go to Anthropic or academia. But who knows.

Everyone knows how chatGPT works.

first Andrej and now Ilya. Talent exodus at OpenAI?

I don't think it's good to lionize people like that, like some kind of tech Übermensch. He neither knows everything about ChatGPT, nor is he the only person there to know a whole lot about it.

I think what is probably very stressful about this space is virtually everyone knows how ChatGPT works. It is not a theoretical leap. It's actually fairly predictable how this shakes out, and OpenAI is pretty vulnerable.

an LLM is a curiosity without user data, anyone with a big silo of data can put out something years behind frontier and still instantly see huge usage. No one wants to go to AI, they want AI to come to them, unless OpenAI can stake a claim in a super novel way they're the Dropbox.

It's not like someone is going to use insider OpenAI knowledge to build an LLM so advanced you switch email, phone, or ERP providers

Last time such a departure led to the creation of Anthropic (Dario Amodei), they even equalled GPT-4's performance

I think it's less that only 4 wizards on earth can create it, and more that hardly anyone wants to.

If Apple, Google, or MS are integrating LLMs, they can't get away with "you can talk to this website". The intersection of "has the cash to train a LLM" and "LLM is itself the entire offering" is very small

No they don’t; if they did Google and Meta would have put out offerings that objectively beat OpenAI. However (barring temporary lapses) they’ve stayed ahead of the curve. Someone who constantly thinks low of their competition or people they hate is bound to fail.

First, "objectively beat" doesn't matter. Comparative LLM performance means more or less nothing without application.

Let's say Google's LLM has slightly poorer reasoning, and you have to be really clear when you tell it to delete your old emails. What are you going to do, go to ChatGPT and have it very eloquently walk you through how to manually delete your emails?

But the idea that Google, the terrifying nation-state, who covetously gobbled up most of the bright minds of a generation, just couldn't fathom RLHF, means you think there is some inherent magic at OpenAI.

I use OpenAI's models a lot, obviously they're great, but whole-thread-as-context as a product is not a product, and Google has people plural who could execute that from first principles.

> It's not like someone is going to use insider OpenAI knowledge to build an LLM so advanced you switch email, phone, or ERP providers

If you mean the other kind of ERP they will definitely do that; it's basically all /r/localllama is about.