Websites these days are often pretty hostile to crawling aside from the Googlebot. Other crawlers are asked not to by the robots.txt, fully blocked, or put through captchas.

https://knuckleheads.club/

https://www.nytimes.com/2020/12/14/technology/how-google-dom...

Good point. Now I think about it Google is actively contributing to that with their Reddit deal which has them blocking all scrapers besides Googlebot, even known legit search engine crawlers like Bingbot.

> Surely the crawling is the easiest part? The AI goldrush has seen dozens of new players gobbling up the entire internet.

The AI goldrush also has sites scrambling to block any crawlers other than a short list of approved search engines. Even before this mess Googlebot running on Google IPs often received special treatment.

Search itself is still a hard problem but one that can be solved independently. Crawling on the other hand requires websites to cooperate and due to Google's size enough won't care to bother with anyone else.