> Webmasters also put highly specific URLs they don't want the world to see in robots.txt. Also not a great idea.

I can confirm; I tried to helpfully put a bunch of addresses which would be pointless for a web crawler to crawl in there (links to the login page, etc), but as soon as you say "don't" somebody's gotta go find out why.

I like the idea of putting gzip bombs at commonly-scanned URLs.