A problem with robots is that they will scan all of a website. Normal user traffic is somewhat focused and is easy to cache. When one robot comes and scans everything at once, it brings a bunch of unpopular pages in the cache, possibly evicting more popular ones in the process. The more popular one will then need to be re-cached as requests come back for it.

If you avoid caching requests by robots, then instead you end up having to go through all the layers of your app, possibly going in the database.

In most situation, I don't think the above matters much. But I can see why it could be a worst case for some stacks.

That's easily detectable and mitigable via a MRU cache, or no caching at all.

That might have been true in 1997, but in 2014 a page every few seconds is so far back in the noise that it's irrelevant. If you are getting problems from an IP it's normally that they are trying 1000's requests per second, and this isn't any halfway sensible spider, it's a dos attack. Robots.txt only stops you if you read and abide by it, and if you are getting a page every few seconds it's almost certain that the domain will not notice.