Google Unfairly Curbs Web Ads and Skews Search, EU Alleges

Re:Google Unfairly Curbs Web Ads and Skews Search, EU Alleges by __b__ on Thursday July 14, 2016 @11:00PM (#12098707)

Thought experiment:

You have a very large site with large amount of content.

You allow Google to crawl your website and store a copy of all your content in its cache.

This amounts to hundreds of pages. That is, many pages of results in the Google search engine.

Imagine for the thought experiment you later lose your website and all the content.

Can you "crawl Google", e.g., site:website.com, to get it back?

No. You will draw a ban.

Google is in some ways like the Internet Archive but with some very peculiar rules.

The Internet Archive encourages users to crawl it.

Is crawling a special privilege that is only reserved for search engines catering to advertisers?

[ Parent | on HN ]

Re:Google Unfairly Curbs Web Ads and Skews Search, EU Alleges by LoSboccacc on Friday July 15, 2016 @03:29AM (#12099412)

google is not your backup service, news at 11

[ Parent | on HN ]

Re:Google Unfairly Curbs Web Ads and Skews Search, EU Alleges by __b__ on Friday July 15, 2016 @10:07AM (#12101018)

The issue is reciprocity as to crawling. There will never be "news at 11" about that.

[ Parent | on HN ]

Re:Google Unfairly Curbs Web Ads and Skews Search, EU Alleges by tedunangst on Friday July 15, 2016 @01:21PM (#12102492)

So ban the googlebot in robots.txt and tell google you'll let them back in when they let you crawl their site.

[ Parent | on HN ]

Re:Google Unfairly Curbs Web Ads and Skews Search, EU Alleges by __b__ on Friday July 15, 2016 @09:23PM (#12104795)

Or allow Googlebot in robots.txt and then crawl Google.

[ Parent | on HN ]