Re:Analyzing One Million Robots.txt Files
by greglindahl on Sunday December 24, 2017 @09:33PM
(#16002319)
That's a best practice for everyone?
[ Parent
| on HN ]
Re:Analyzing One Million Robots.txt Files
by walshemj on Monday December 25, 2017 @10:01AM
(#16004302)
yep if your first Line is a disallow and you have a text file with a BOM it will be ignored.
Re:Analyzing One Million Robots.txt Files
by greglindahl on Monday December 25, 2017 @02:19PM
(#16005340)
I was more thinking that most webmasters don't use editors and OSes where a BOM might be emitted.