Search engines seem to be stuck between serving two roles: 1. An easily accessible directory of mainstream information, and 2. A specialized tool to find the diamond in the rough. It seems like it has to be a tradeoff, it can't serve both roles equally well.