News for hackers. Stuff that matters.

A comment on Google’s robots.txt parser is now open source. Parent comment

Re:Google’s robots.txt parser is now open source by inlined on Wednesday July 10, 2019 @12:48AM (#20399216)
5xx means that the server made a mistake. 4xx means that the caller made a mistake. Sending a request to a GONE url is canonically classified as a “user” or sender error.
[ Parent | on HN ]

There are no comments on this comment yet.