Re:The Scraping Problem and Ethics
by owlmanatt on Thursday May 08, 2014 @10:01AM
(#7715287)
> ii) Should search engines pay the scraped sites if they are charging to access their indexed data? probably some of the scraped sites has a specific license forbidding the search engine to sell their information in any way.
Any reputable search engine will respect robots.txt.
Any reputable search engine will respect robots.txt.