News for hackers. Stuff that matters.
Re:The HTTP status code for a web server's default “hello” f... by Chaosvex on Sunday July 09, 2023 @03:10AM (#36652332)
Definitely overcomplicating things there. robots.txt already exists.
Re:The HTTP status code for a web server's default “hello” f... by hughesjj on Sunday July 09, 2023 @04:26AM (#36652690)
If they haven't set up their landing page I doubt they configured a robots.txt
Re:The HTTP status code for a web server's default “hello” f... by Chaosvex on Sunday July 09, 2023 @08:10AM (#36653805)
This is clearly talking about default behaviours, so it'd already be there as standard. It's no different than expecting a different status code for the purpose of requesting crawlers not to index.
Re:The HTTP status code for a web server's default “hello” f... by weird-eye-issue on Sunday July 09, 2023 @11:21AM (#36655202)
robots.txt deindexing a site by default is not necessarily the right choice. Lots of people are clueless about it and would never even think to change it when going live.