News for hackers. Stuff that matters.
Re:If Google sucks then why is everyone still using it? by derefr on Saturday March 19, 2022 @01:58AM (#30731512)
If a site allows crawling by at least one public-Internet spider, is there any legal protection it has against other crawlers who choose to ignore robots.txt and crawl anyway? Because I feel like that's exactly what Google would do here, as long as it wasn't literally illegal for them to do so.
Re:If Google sucks then why is everyone still using it? by Sirened on Saturday March 19, 2022 @02:20AM (#30731620)
It's kinda unclear at the moment, but it's working its way through the courts! See HiQ Labs v. LinkedIn [1] in which HiQ was scraping public profiles and was blocked from doing so by LinkedIn. This made its way to court and the 9th circuit ruled they were allowed to scrape but SCOTUS later rejected the decision and sent it back. So—murky!

[1] https://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn

Re:If Google sucks then why is everyone still using it? by sfe22 on Saturday March 19, 2022 @06:45AM (#30732739)
You can still block googles ip’s manually