Because classifying huge amounts of content is highly non-trivial. It's not like there are a million employees sifting through every spidered resource and manually deciding whether it should be indexed.