Google’s robots.txt parser is now open source

Re:Google’s robots.txt parser is now open source by C1sc0cat on Monday July 01, 2019 @12:09PM (#20326772)

Robots stops a page from being crawled the noindex tag stops it getting into the index.

Google is also slow to honour 404 and drop pages which can hang around for ages, Bing is much faster to remove 404 pages.

[ Parent | on HN ]

Re:Google’s robots.txt parser is now open source by rafaelm on Monday July 01, 2019 @12:15PM (#20326831)

Usually, creating "410 Gone"[0] response for the URL and running the URL through the URL Inspection Tool [1] can help make things a bit faster. But yeah, it does take a while to get these 404s removed.

[0] https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/410 [1] https://support.google.com/webmasters/answer/6065812?hl=en

[ Parent | on HN ]

Re:Google’s robots.txt parser is now open source by inlined on Monday July 01, 2019 @02:09PM (#20328081)

That distinction exists in many systems. E.g. for cloud events, 404 is considered with skepticism because it could be a race condition in provisioning or transient issue whereas 410 requires data streams to be cut off.

[ Parent | on HN ]

Re:Google’s robots.txt parser is now open source by C1sc0cat on Tuesday July 02, 2019 @05:42AM (#20333571)

Then you should serve 503

[ Parent | on HN ]

Re:Google’s robots.txt parser is now open source by inlined on Wednesday July 10, 2019 @12:48AM (#20399216)

5xx means that the server made a mistake. 4xx means that the caller made a mistake. Sending a request to a GONE url is canonically classified as a “user” or sender error.

[ Parent | on HN ]