Re:Tell HN: HN Was Down
by Bender on Wednesday December 17, 2025 @02:42PM
(#46304495)
Most crawlers have no concept of what that is. They will follow links to this site and then follow links out of this site even after being told not to [1]. The majority of crawlers follow zero rules, RFC's, etc... The few platforms that do follow standards and rules are akin to a law abiding citizen in Mos Eisley.
[1] - rel="nofollow"