News for hackers. Stuff that matters.

A comment on Web Scraping in 2016. Parent comment

Re:Web Scraping in 2016 by tedunangst on Tuesday August 23, 2016 @06:22PM (#12348261)
Google will put forbidden pages in its index. It doesn't scrape them. (The URL to the page exists even without visiting the page.)
[ Parent | on HN ]

There are no comments on this comment yet.