News for hackers. Stuff that matters.
Re:Analyzing One Million Robots.txt Files by tomcam on Sunday December 24, 2017 @12:06PM (#16000281)
Surprisingly interesting post that goes into history of robots.txt and details how it is not, in fact, a W3 standard or legal requirement
Re:Analyzing One Million Robots.txt Files by walshemj on Sunday December 24, 2017 @02:57PM (#16001043)
does seem obsessed with the non standard crawl delay interestingly it doesn't mention those robots.txt files that have a BOM which can stop them working - which is why best practice is to have a comment of a blank line at the top of your robots.txt
Re:Analyzing One Million Robots.txt Files by greglindahl on Sunday December 24, 2017 @09:33PM (#16002319)
That's a best practice for everyone?
Re:Analyzing One Million Robots.txt Files by walshemj on Monday December 25, 2017 @10:01AM (#16004302)
yep if your first Line is a disallow and you have a text file with a BOM it will be ignored.
Re:Analyzing One Million Robots.txt Files by greglindahl on Monday December 25, 2017 @02:19PM (#16005340)
I was more thinking that most webmasters don't use editors and OSes where a BOM might be emitted.