I also have a bunch of hobby websites, and I agree that bing crawls a lot more than Google and sends me a lot less traffic. And Yandex and Baidu have very little English-speaking usage, so they aren't sending me much traffic.
If I see that a crawler is sending me any traffic at all, I will accept that, but if the amount of traffic is zero, I put them in robots.txt and in the IP block list. Although I try hard to make sure my images are all clean creative common images, I block the robots that are there to find copyrighted images because these ones exist to give me nothing but trouble.
I've never heard of these bots before. Where did you hear about them, and how do you block them?
EDIT: chrsstrm mentions something below
Google is the only crawler I see using the protocol on my site. It does make Google updates occur within minutes, so that alone is reason to implement push.
- If it's cached content, there's no limit
- If it's non cached content and it is a bot, direct to the pool of servers assigned to uninteresting bots
- If it is an interesting bot (Google, bing, a couple others), use instead a pool of servers for interesting bots
Some bots are not only non-traffic generating, they are absurdly aggressive, consuming 99% of server resources for the hour or so needed to pull the content, regular site users be damned. A configuration like this is reasonably low maintenance while limiting the damage by bots.
I run a small wiki that gets just a few thousand human hits a day. But according to the server logs 90% of server hits are crawlers and spambots, so I'm using 10 times the resources I really need to serve customers.
I finally resorted to blocking entire data centers and companies that crawl constantly but send no traffic.
I feel like search engines should crawl websites in proportion to the traffic they send. For example, Yandex, Baidu, and Bing were all crawling my website hundreds or thousands of times a day, but never sending a single visitor (or in the case of Bing sending single-digit visitors). It's an absurd waste of resources, so I blocked them completely.