What is the exact point that a crawler needs to be decoupled (at the corporate level) from it's front end? Is it measured by MAUs? Pages scraped? Or is it going by "it can be decoupled, so it should".
The extreme version of your argument would have companies own the IP to a particular for loop's iterator variable name.
A search engine is a number of components - e.g. a crawler, an index, and a user interface.
Why should one company be able to own all three of these? Why not just one? Perhaps crawling should only be able to be sold as a wholesale commodity.