Google no longer just "crawls" in the sort of general way that crawlers work to populate its search index, but also "scrapes", - that is to say they seek out specific information from some websites in order to display their data in the Google answer box, instead of directing you to their website to get that information.

While not illegal, I think it is unethical and breaks the symbiosis that their search engine had with the web.

I don't think it could be solved globally; local jurisdictions like the EU will need to deal with this issue sooner or later. I think American companies can try to lobby against it too, because Google abuses its enormous power to eliminate competition in various markets (product price comparison etc.). There are many American companies operating in this space. Google's strategy is to steal it slowly piece by piece and it looks like they're getting more and more bold.

Today, Google even crawls, renders and indexes URLs sent from Chrome browsers via dial-up IPs using regular browser User-agents, so... no. Other than that, Googlebot seems to obey most of the time.

I don't blame them. Cloaking couldn't be detected otherwise.

I think i read somewhere that Google crawlers sometimes pretend to be real browsers to prevent you from displaying a different page the crawler. So, no.

I guess he never read the reply to the question he posted