Data pollution isn't the problem it is made out to be. You can even use a model trained on garbage data to relabel the original set and retrain.

Google of all companies is sitting on exaflops of TPU capacity, they can retrain models larger than anyone else, faster than anyone else.

I agree, but it would probably require to use LLMs to filter or augment the raw text. They can afford it.

Their data may not be that much better; however Deepmind is still one of the best institutions that study AI fundamentals. OpenAI works hard and focused towards products, but the academic research is far better from Deepmind. So since Google hasn't been trying to put any AI into products really at all for the past 10 years, they're going to have to push a little bit to make the public facing interfaces and put polish on things, but the underlying models they have made are far better than OpenAI (just not the ones made publicly accessible to anyone).

They have no model on par with GPT-4.

I would heavily disagree with that - the only reason that you think that is that GPT-4 is available to the public for cheap, whereas Deepminds models have not been released publicly.

For websites that blocks all robots.txt-compliant bots except Google, Bing and some, Googlebots can crawl but CommonCrawl bots can't