> Surely the crawling is the easiest part? The AI goldrush has seen dozens of new players gobbling up the entire internet.

The AI goldrush also has sites scrambling to block any crawlers other than a short list of approved search engines. Even before this mess Googlebot running on Google IPs often received special treatment.

Search itself is still a hard problem but one that can be solved independently. Crawling on the other hand requires websites to cooperate and due to Google's size enough won't care to bother with anyone else.