News for hackers. Stuff that matters.

A comment on Only Google is really allowed to crawl the web. Parent comment

Re:Only Google is really allowed to crawl the web by bigwavedave on Friday March 26, 2021 @01:02PM (#26594394)
> Is it legal for a government entity to issue a robots.txt like that?

I may be wrong (this isn't my area), but I was under the impression that robots.txt was just an unofficial convention? I'm not saying people should ignore robots.txt, but are there legal ramifications if ignored? I'm not asking about techniques sites use to discourage crawlers/scrapers, I'm specifically wondering if robots.txt has any legal weight.

[ Parent | on HN ]
Re:Only Google is really allowed to crawl the web by banana_giraffe on Friday March 26, 2021 @11:02PM (#26599535)
Yep, it's defined at https://www.robotstxt.org/

Looks like Google is trying to turn it into an RFC though.

[ Parent | on HN ]