News for hackers. Stuff that matters.

A comment on 9th Circuit holds that scraping a public website does not violate the CFAA [pdf]. Parent comment

Re:9th Circuit holds that scraping a public website does not... by skybrian on Monday September 09, 2019 @10:45PM (#20924642)
Google respects robots.txt, so it might be hard to prove that they are accessing websites without their implied consent.

Also, their own robots.txt contains "Disallow: /search". So, there is arguably no inconsistency, either.

But, what does this new ruling mean for robots.txt?

[ Parent | on HN ]
Re:9th Circuit holds that scraping a public website does not... by dclusin on Tuesday September 10, 2019 @01:21AM (#20925239)
I think OP is getting at the nature of the relationship is kinda imbalanced. Consider basically most of their website is off limits: https://www.google.com/robots.txt
[ Parent | on HN ]
Re:9th Circuit holds that scraping a public website does not... by nojvek on Tuesday September 10, 2019 @02:06PM (#20931245)
you can have your website offlimits from google by having a robots.txt too. What's the problem? People willingly want google to index them so they appear in search results and google can send them traffic.
[ Parent | on HN ]