News for hackers. Stuff that matters.

A comment on Google’s robots.txt parser is now open source. Parent comment

Re:Google’s robots.txt parser is now open source by rafaelm on Monday July 01, 2019 @12:15PM (#20326831)
Usually, creating "410 Gone"[0] response for the URL and running the URL through the URL Inspection Tool [1] can help make things a bit faster. But yeah, it does take a while to get these 404s removed.

[0] https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/410 [1] https://support.google.com/webmasters/answer/6065812?hl=en

[ Parent | on HN ]
Re:Google’s robots.txt parser is now open source by inlined on Monday July 01, 2019 @02:09PM (#20328081)
That distinction exists in many systems. E.g. for cloud events, 404 is considered with skepticism because it could be a race condition in provisioning or transient issue whereas 410 requires data streams to be cut off.
[ Parent | on HN ]
Re:Google’s robots.txt parser is now open source by C1sc0cat on Tuesday July 02, 2019 @05:42AM (#20333571)
Then you should serve 503
[ Parent | on HN ]
Re:Google’s robots.txt parser is now open source by inlined on Wednesday July 10, 2019 @12:48AM (#20399216)
5xx means that the server made a mistake. 4xx means that the caller made a mistake. Sending a request to a GONE url is canonically classified as a “user” or sender error.
[ Parent | on HN ]