What are the chances that their copy is more useful than CommonCrawl? It’s been the target of SEO for decades - are they even capable of separating the good content from the crap anymore?

Google may already be terminal

Data pollution isn't the problem it is made out to be. You can even use a model trained on garbage data to relabel the original set and retrain.

Google of all companies is sitting on exaflops of TPU capacity, they can retrain models larger than anyone else, faster than anyone else.

I agree, but it would probably require to use LLMs to filter or augment the raw text. They can afford it.

Their data may not be that much better; however Deepmind is still one of the best institutions that study AI fundamentals. OpenAI works hard and focused towards products, but the academic research is far better from Deepmind. So since Google hasn't been trying to put any AI into products really at all for the past 10 years, they're going to have to push a little bit to make the public facing interfaces and put polish on things, but the underlying models they have made are far better than OpenAI (just not the ones made publicly accessible to anyone).

They have no model on par with GPT-4.

I would heavily disagree with that - the only reason that you think that is that GPT-4 is available to the public for cheap, whereas Deepminds models have not been released publicly.

For websites that blocks all robots.txt-compliant bots except Google, Bing and some, Googlebots can crawl but CommonCrawl bots can't

> The doc is too short and not puffy to be a normal google doc.

Loads of internal position papers read exactly like this. I've written docs that write like this.

They also have "all of youtube" when it comes to video based learning. And GCS and Gmail, Google's moat is actually enormous.

The points raised in the paper are that

1) algorithmic improvements beat processing capacity. One of the examples in the paper was that $100 in training compute budget was enough to get within 2% of Google Bard performance

2) data quality (even for "tiny" data sizes) beats dataset size in improvements

Hence both aspects of the moat of large companies (dataset size and available compute) matter less than you'd expect.

In contrast the advantage of "open source"/academics/"the market", the many warm bodies trying out new strategies, yields improvements at a rapid pace that Google, even with a huge team, cannot match.