Bell labs and Xerox PARC did great impactful work long after their parent companies were relevant and still do .

The fact Google did the initial work on transformers but only OpenAI was able to productize is an indictment of their stagnation more than an achievement.

Bell Labs and Xerox PARC were great improvements on their successors.

Or they recognised the fundamental problem which is that LLM's will kill the motivation for people to upload new information to feed the LLM's with.

If websites can't earn a living through ads because LLM's don't send them any traffic anymore, where do the LLM's ingest new information from?

The web continues to deteriorate because of this and nobody has solved this problem.

I’m pretty sure they were just terrified of the PR impact of another “humans classified as gorillas” type incident.

And they ended up with "put elmers glue on pizza" type incident.

AI is messy

It’s fundamental, IMO.

LLMs have to have something ‘true’ (or at least statistically probable) to validate against for training.

Taking the output from it and feeding it back into itself is, essentially, an Ouroboros. Or like locking a human in solitary.

The more/longer it happens, the more deranged it’s going to get.

I agree that they recognized the fundamental problem but I have different speculation as to what the problem is: the problem must be it's technologically too boring and obviously not profitable to them.

It's culmination of decades of NLP researches, but it's also just East Asian predictive text percussively adapted for variable word length language. It must've been clear to whoever core people that it's fundamentally a sluggish demo or at most a cheap outright-sold commodity product.

IMO they would be right unless someone finds a reason that LLMs can't ever be self hosted in the way Google Search can't be. They must have just saw by the way it is that LLM is at best a regularization engine for that magic box.

> only OpenAI was able to productize

What do you mean? Anthropic and Google both have widely used products based on transformers.

OpenAI is the market leader by far with the most name recognition. Google was the last to market. Its initial release of Gemini was a total flop because of the meme "Use elmer's glue on pizza to keep the cheese on". It has finally become more consistent, and it manages to compete with other models though I never see anyone recommending Gemini first.

All of these companies are in the red, but OpenAI has the most revenue.

This is a bit of a tangent, but I don't think OpenAI's brand is all that durable. You can see that Perplexity.AI has been gaining rapidly. At this point they have half as much search traffic as OpenAI:

https://trends.google.com/trends/explore?date=today%205-y&ge...

You don't search for openai. You go to chat(gpt).com

Wouldn't that also apply to Perplexity.AI?

As I recall, the timeline was:

2017, seven Google employees invent the transformer architecture and publish a paper. Google's investing heavily into ML, with their own custom 'TPU' chips and their 'Tensorflow' ML framework.

2019ish, Google has an internal chatbot they decide to do absolutely nothing with. Some idiot tells the press it's sentient, and they fire him.

2022, ChatGPT launches. It proves really powerful, a product loads of individuals and businesses are ready to pay for, and the value of the company skyrockets.

2023, none of the seven Transformer paper authors are at Google any more. Google rushes out Bard. Turns out they don't have a sentient super-intelligence after all. In fact it's badly received enough they end up needing to rebrand it a few months later.

Classic tortoise-and-hare situation - Google spent 5 years napping, then had to sprint flat out just to take third place.

Except tortoise is supposed to win. Maybe we just haven’t given it enough time?

Google is the hare, in GP's comment.

For the tortoise to win in technology it needs to be dedicated to relentlessly polishing and improving something over a long period to make the best product experience. Those aren't traits I particularly associate with Google unfortunately.

Do you not know the story? The hare is the one that naps and wastes time instead of just going full out and winning from the start.

Have you ever listened to what Lemoine said? Sure, we have no proof and he's under NDA so probably no documentation that can be scrutinized. But still, his alleged chats were chilling in some ways. They probably didn't except him to go public and so they had to spend years nerfing their chat bot before launching it as a product and that's why it sucks: They're too careful and have too much to lose in bonuses. Google will probably lose some market share over the next few years before they're getting nervous to put someone with a longer leash into the CEO seat.

I recall this particular person seeming like a bit of a crackpot on internal forms before (and for reasons unrelated to) the Lamda chatbot. I didn't know him personally and don't even remember the details anymore but it made an impression that wasn't dispelled by his reaction to a new model passing the turing test.

> just to take third place.

At worst, Google is in a tie for first.

It is easier to judge revenue or market share than technical quality of the models itself objectively , they are relatively close to each other functionally .

In the market, I would say both Anthropic and openAI have been able to do that much better than traditional big tech including Google.

Google has very research division but productising those inventions is the problem.

It was used in Google translate, and BERT was incorporated into search in 2019, though I don't think it was a clear win for search, I feel like I started having to add exact quotes to everything technical/programming around then.

One thing I don't understand is google has so much metadata on search sessions to RLHF their search results.

E.g. when I start a search session to solve a programming problem (before llms), I will continually search different terms to get to my solution webpage. Then stop. This session metadata and the path I took is highly significant data that can be used to help llms recognise what research itself looks like.

Not RLHF, but my understanding was they heavily use that data and it was a big part of their moat, part of why competitors wanted to clone their results because they couldn't derive as good of quality from the web alone (Microsoft used the bing toolbar to clone them in the 2010s).