When I worked in newspaper web tech, I did an analysis of our front end traffic, grouped by IP ranges, user agent claims, and so on. I found that the majority of our bandwidth was spiders and bots. The majority of our hits were also robots. I couldn't do anything about that without big spending with our cache vendor to prevent it.