Google’s robots.txt parser is now open source

Re:Google’s robots.txt parser is now open source by inlined on Monday July 01, 2019 @02:09PM (#20328081)

That distinction exists in many systems. E.g. for cloud events, 404 is considered with skepticism because it could be a race condition in provisioning or transient issue whereas 410 requires data streams to be cut off.

[ Parent | on HN ]

Re:Google’s robots.txt parser is now open source by C1sc0cat on Tuesday July 02, 2019 @05:42AM (#20333571)

Then you should serve 503

[ Parent | on HN ]

Re:Google’s robots.txt parser is now open source by inlined on Wednesday July 10, 2019 @12:48AM (#20399216)

5xx means that the server made a mistake. 4xx means that the caller made a mistake. Sending a request to a GONE url is canonically classified as a “user” or sender error.

[ Parent | on HN ]