Re:The HTTP status code for a web server's default “hello” f...
by Chaosvex on Sunday July 09, 2023 @03:10AM
(#36652332)
Definitely overcomplicating things there. robots.txt already exists.
[ Parent
| on HN ]
Re:The HTTP status code for a web server's default “hello” f...
by hughesjj on Sunday July 09, 2023 @04:26AM
(#36652690)
If they haven't set up their landing page I doubt they configured a robots.txt
Re:The HTTP status code for a web server's default “hello” f...
by Chaosvex on Sunday July 09, 2023 @08:10AM
(#36653805)
This is clearly talking about default behaviours, so it'd already be there as standard. It's no different than expecting a different status code for the purpose of requesting crawlers not to index.
Re:The HTTP status code for a web server's default “hello” f...
by weird-eye-issue on Sunday July 09, 2023 @11:21AM
(#36655202)
robots.txt deindexing a site by default is not necessarily the right choice. Lots of people are clueless about it and would never even think to change it when going live.