News for hackers. Stuff that matters.
Re:Wikipedia's robots.txt by bawolff on Tuesday July 16, 2019 @08:00PM (#20455114)
Its because the robots.txt file is created by combining a static file ( https://github.com/wikimedia/operations-mediawiki-config/blo... ) with an on-wiki page so that different language wikipedias can manage the file themselves - https://en.wikipedia.org/wiki/MediaWiki:Robots.txt

In wikisyntax, starting a line with a space ensure's that its in a pre tag, so this was a way to make the onwiki page show the entire thing in a pre tag and not use normal formatting. This seems to be broken by actually starting a literal pre tag inside the first line, but I'm pretty sure this used to work as a way to display the entire thing in a pre tag.

For the curious, the code generating the robots.txt file is at https://github.com/wikimedia/operations-mediawiki-config/blo...

There are no comments on this comment yet.