A problem with robots is that they will scan all of a website. Normal user traffic is somewhat focused and is easy to cache. When one robot comes and scans everything at once, it brings a bunch of unpopular pages in the cache, possibly evicting more popular ones in the process. The more popular one will then need to be re-cached as requests come back for it.

If you avoid caching requests by robots, then instead you end up having to go through all the layers of your app, possibly going in the database.

In most situation, I don't think the above matters much. But I can see why it could be a worst case for some stacks.