"Please note - This e-mail and any files transmitted with it may contain confidential or privileged information which is protected by legislation, copyright & the code of practice covering personal information. It is intended solely for the use of the individual or entity to whom it is addressed. If you have received this e-mail in error, please notify the originator. Any disclosure, copying, alteration, distribution, publication or the taking of action in reliance on the contents may be an offence."
as a random example. Bonus points for using it it in a footer, where almost by definition, you've read the email before seeing it.
I'm sure there's some legal justification for doing so, but it still strikes me as pretty stupid.
Well, robots.txt is a simple text file that politely asks robots to not spider parts of your website.
It is completely voluntary and unenforceable, but works very well.
It is completely voluntary and unenforceable, but works very well.
Ha! It works very well: for Google, and Yahoo. They use informative user-agents, and respect robots.txt directives. They have to, they're large corporations, with shareholders.Everybody else ignores it. Why would they listen?
Live example:
216.55.185.45 - - [11/Sep/2011:06:50:56 -0400] "GET / HTTP/1.0" 200 4835 "-" "Mozilla/5.0 (Windows NT 6.1; rv:5.0) Gecko/20100101 Firefox/5.0"
216.55.185.45 - - [11/Sep/2011:06:50:57 -0400] "GET / HTTP/1.0" 200 4835 "-" "Mozilla/5.0 (Windows NT 6.1; rv:5.0) Gecko/20100101 Firefox/5.0"
216.55.185.45 - - [11/Sep/2011:06:50:57 -0400] "GET /blog/ HTTP/1.0" 200 114535 "-" "Mozilla/5.0 (Windows NT 6.1; rv:5.0) Gecko/20100101 Firefox/5.0"
216.55.185.45 - - [11/Sep/2011:06:51:01 -0400] "GET /blog/archives/2010/01/12/creating_an_account_on_the_scp_wiki_is_like_pissing_glass/ HTTP/1.0" 200 10490 "-" "Mozilla/5.0 (Windows NT 6.1; rv:5.0) Gecko/20100101 Firefox/5.0"
This is all the traffic from this IP, with no lines skipped. You might notice that it doesn't request CSS, or any of the linked images, or favicon.ico. Nor does it populate the referral header fields, which of course it would be doing if it was actually Firefox, and there was actually a human clicking on these links. It doesn't even bother requesting a robots.txt, which I don't have anyway.[1] Do a whois check on the IP, and we get: Codero CODERO1999A (NET-216-55-176-0-1) 216.55.176.0 - 216.55.187.255
Codero's a dedicated server host. This is a spambot, looking for email addresses. Visit the IP in a browser, and you see a site selling fake Tiffany jewelry.