There are some factors that might lead to this lone of though.

- it seem a lot of web developer / product manager either do not know or do not care about robots.txt

- some web application are so badly optimized, that some of them are not able to handle more than 1 hit per second at a sustained rate, which admittedly have worked fine so far. But crawlers are persistent, causing the normal crawling activity to cause denial of service for normal users.