Reddit, newspapers, Facebook, Twitter, are all locked down or heavily restricted. I am sure there is still a burgeoning small web, but it is increasingly hard to find. Would many consumers notice/care if your search engine only surfaced pages from the top 1000 highest ranked domains? You would actually decrease the amount of SEO garbage you hit from Stackoverflow clones, Amazon listicles, or Grandma's cookie recipe #9381.