> Google's dominant market position was achieved through market competition not through impairing the ability of smaller companies to enter markets.

Googlebot was allowed to its thing pretty freely back in the day on the assumption that Google will remain a search company only and will not step on the toes of the companies whose websites it was parsing. That proved out not to be true.

What's more, if today I want to parse Google's web properties (let's say its GoogleMaps data) the same way that Googlebot used to parse the websites of the companies I used to work for I'll get banned almost immediately.