A lot of robots.txt exist to help crawlers automatically know which URLs are safe and which ones are mutable (i.e. the sign up link).

You should only ignore the robots.txt if you are very careful about what you are doing and have a very good reason for doing so.