News for hackers. Stuff that matters.

A comment on Bypassing website anti-scraping protections. Parent comment

Re:Bypassing website anti-scraping protections by avip on Friday June 01, 2018 @03:08PM (#17208887)
Google indexers respect robots.txt, so there goes the irony.
[ Parent | on HN ]
Re:Bypassing website anti-scraping protections by p49k on Saturday June 02, 2018 @02:01PM (#17215612)
They also provide something of value to the operators of the websites they scrape, namely search traffic.
[ Parent | on HN ]
Re:Bypassing website anti-scraping protections by pbhjpbhj on Friday June 01, 2018 @03:37PM (#17209264)
I've heard that they [sometimes] visit but don't index, is that true?
[ Parent | on HN ]