News for hackers. Stuff that matters.

A comment on Only Google is really allowed to crawl the web. Parent comment

Re:Only Google is really allowed to crawl the web by banana_giraffe on Friday March 26, 2021 @11:02PM (#26599535)
Yep, it's defined at https://www.robotstxt.org/

Looks like Google is trying to turn it into an RFC though.

[ Parent | on HN ]

There are no comments on this comment yet.