Search engines are really a relic from back when everyone had their own webpage. Now everyone posts their content on a handful* of sites - Medium, Reddit, StackOverflow. I think what we need is an inversion of the search engine model... instead of a web crawler scouring the web for new pages, these handful of companies (Medium, Reddit, thousands more) should register with the search engines and push sitemaps regularly. This is the only way to level the playing field with Google, since the internet is so vast now that only Google/Microsoft/Yandex can afford the server/bandwidth fees to crawl it.

*- and by "handful" I mean tens of thousands of sites, which is relatively small.