News for hackers. Stuff that matters.

A comment on Show HN: CloudScrape – Cloud-based web scraping platform. Parent comment

Re:Show HN: CloudScrape – Cloud-based web scraping platform by chinathrow on Sunday October 18, 2015 @04:21AM (#10407585)
Google also adheres to the robots.txt standard. Most of the scrapers I block don't.
[ Parent | on HN ]
Re:Show HN: CloudScrape – Cloud-based web scraping platform by jumperjake on Sunday October 18, 2015 @12:01PM (#10408569)
Not correct. Google will completely ignore the rules in robots.txt if it deems it acceptable. I think there's a link to this somewhere in this comment page.
[ Parent | on HN ]
Re:Show HN: CloudScrape – Cloud-based web scraping platform by chinathrow on Sunday October 18, 2015 @02:41PM (#10409212)
They do not index the content but might add the URL, correct. You can have a meta noindex present and they won't index even the URL.
[ Parent | on HN ]