News for hackers. Stuff that matters.
Re:Only Google is really allowed to crawl the web by TheAdamAndChe on Friday March 26, 2021 @12:19PM (#26593880)
Are there any actual repercussions for just ignoring robots.txt?
Re:Only Google is really allowed to crawl the web by asciident on Friday March 26, 2021 @01:24PM (#26594663)
There is if you are doing it for work. For example, your company could get sued if you are found using that data and ignoring the ToS. If you are a public figure, you could get your name tarnished as doing something unethical or the media may call it "hacking". If you are rereleasing the data then you risk getting a takedown notice.
Re:Only Google is really allowed to crawl the web by eertami on Friday March 26, 2021 @09:32PM (#26599034)
robots.txt is not a terms of service. Even if it was, it wouldn't be enforceable for a public website. You would need to prove that a web crawler is maliciously causing disruption to your service, and that is not easy.
Re:Only Google is really allowed to crawl the web by asciident on Friday March 26, 2021 @10:26PM (#26599315)
All it takes is your company execs or lawyers to be afraid of a stern letter, and ask you to cancel your project. If you're violating their robots.txt, you're probably violating their terms of service that's hidden somewhere. And your company doesn't want to risk having to pay hundreds of thousands to fight a court case. There's also venues besides courts for them to attack you, like contacting the publishers or hosting platforms for your derivative works. It's a chilling effect.

And I'm not making this up. This kind of stuff has happened to me many times.

Re:Only Google is really allowed to crawl the web by 5560675260 on Friday March 26, 2021 @12:39PM (#26594122)
Your crawler's IP might get banned, eventually.
Re:Only Google is really allowed to crawl the web by the_dege on Friday March 26, 2021 @12:54PM (#26594298)
Sometimes website admins will also try to report your ips to the service provider as a source of attacks (even if not true).
Re:Only Google is really allowed to crawl the web by DocTomoe on Friday March 26, 2021 @05:37PM (#26597258)
Given how often I've had misbehaving crawlers slow own servers in the early 2000s, I do not see how a crawler that disobeys robots.txt is not an attempted attack.