Maybe because libraries are not private compaines? I mean, you mention competitors not having the revenue to crawl an increasingly complex web. But how else do you see them being able to facilitate this without a similar advertising strategy to Google?

I just have little patience for complaints about free products like Facebook, Google, Twitter etc. They cost you nothing but time and attention - both of which you have complete control over.