> From a moral point of view: robots.txt states an intent, and intentionally ignoring that is not a nice thing.

As a lawyer asked inane questions by friends, a common answer is: "I am no sure if its legal to do, but I'm absolutely sure you're a asshole if you do it."