It would just take at astronomical amount of capital to challenge Google at generic Internet search, and the few organizations with that capital are pretty clear they don't think it's a good bet. For example, Amazon is quite content just being the place you want to go to first for the things you want to buy, as that is obviously the most lucrative area.

Even in niche areas, I think it's still an enormous task to challenge Google. Despite the fact that 90% of my searches for "WTF does this stacktrace mean?" end up on StackOverflow, I invariably start my search on Google because every now and then it gives me useful non-StackOverflow tidbits, and I know I can always drill in more with specific tags on StackOverflow later.

It's telling, that a Google search scoped to a particular web site (xyz site:website.com) almost always produces better results than that web site's actual built-in search engine. If a web site that knows in detail its own content, its own users, and its own domain, still can't make their built-in search better, how does anyone else have a chance?

Google was epic when it started but quickly ended up indexing it self in a near closed loop. Everyone else was not important, it was all about the picture in front of them that was really just a mirror. Everyone stopped making home pages, forums and weblogs because it just doesn't make sense to do it.

Shit, people looking to buy something talking among their own before doing the purchase? KILL IT WITH FIRE!

that becomes a big can of worms very quickly, and starts to put into perspective some of googles other most successful verticals: NLP, Translation, Maps ("Near me"). Of course a lot of the verticals are also a generic means to get value out of all the server resources and experts required to operate the engine and then additional related branches to those verticals (email -> drive -> docs)

You only search whitelisted sites, which removes a lot of the SEO and blogspam from the results, and makes the index is wildly smaller. Search is free, but users can also pay a cheap subscription for the ability to flag sites as spammy, submit sites they think are missing, and submit queries the site has trouble with. Someone manually goes through the top N most-repeated submissions each day (early on this is all of them) and determines whether to tweak the whitelist.

Because your initial audience has to be people who are very enthusiastic about better search results, you should identify what they're interested in and focus your efforts on initial whitelists to things they care about. That might be technical questions (MDN StackOverflow, every random framework's docs site), or maybe whatever it is journalists would want to have access to (I'm not qualified to answer).

I don't think VC-encouraged hypergrowth is possible with the above strategy, but you could probably get to ramen profitable in a reasonable time frame, with slow growth ever after.

The fate of runnaroo seems to provide evidence to the contrary.

I understand your intent with that domain-specific example but surprisingly, that reasonable intuition sometimes doesn't work...

E.g. for Amazon product search, I often deliberately use Google's engine to find the Amazon page because Amazon's own search can't do it. See example screenshots in my previous comment about that: https://news.ycombinator.com/item?id=27539199

Curious what you think these are. I prefer Bing's UI to Google's but aside from that Bing is (IMO) a dumpster fire that quite literally makes me feel negligent (towards myself) when I use it - am I missing something?

In the past, Google search results will point to some good websites you never heard of and are very low traffic. But nowadays, the first couple pages are all popular websites with huge traffic even when they are low quality on the content. You can only find low traffic but good content websites in comments from social sites like reddit and HN.

So it seems to me, to build a usable search engine, you only need to index most popular websites.

I would disagree actually. I think the majority of searches people do and the solutions they would be satisfied with require only indexing major sites. Create a better UX attached to PageRank of the top 10-100k+ Alexa ranked sites and you'd have a competitor that could focus on other improvements.

At the bottom you could search Google or DDG if you want more niche results.

I.e., optimizing UX of the p90 searches would be enough for people to move and the p90 of searches are happy with a pretty narrow range of sites. (You could insert value prop of: no ads, no tracking, and crowd-sourced refinement -- i.e., realtime upvote/downvoting -- of the top results, previews, caching, more knowledge graph stuff on the right side, deeper metadata processing of top sites (i.e., avoiding the need to 'site:*'), or some other revolutionary UX like a one sentence hypothesis solution of exactly what the person is asking for again upvoted/downvoted for refinement -- Google does this but there is a lot of room for improvement). It's fun to think about. There is an order of magnitude in both precision and accuracy that can be improved for huge swaths of common questions people ask on Google.

Agree we need a coop of search engines, but I'll add that probably an org like mozilla is best suited for it, that way it can stay non-profit and recycle any profits back into R&D. personally I believe most of these solid business plan companies can be replaced by a simpler open-source version as IMO a lot of their energy is spend in securing revenue lines & profits.

Mozilla is probably in the best position to do it, I'm just not sure they actually would do it in a way that appealed to consumers.

This doesn't address why MSFT can't compete with Google. If you solve that problem, then you really get to the heart of the solution.

Isn't this exactly the reason why one should do it, as Google is the essence of the online advertising industry and this very regulation could threaten its existence?

>I think there is massive opportunity for domain specific search engines though

Challenge with niche search engine is discoverability. Imagine you had 10 of the greatest vertical search engines available today - how do you remember which one to use and how? The beauty of a general web search is that it is an easy to remember entry point to all of what is out there.

> You then potentially move towards a building “meta” search engines

Ever wondered why is this called meta search engine, and an email software that connects to multiple email servers is called email client and not "meta email". "Search engine client" sounds like a more appropriate name.

The challenge comes from the massive investments in turning that data into a good search engine and cutting through spam. That is the investment I think Google has scaled over time. Secondly, mind share; the moat Google has is immense, although I think DuckDuckGo and Bing has shown given enough time it is possible to get mindshare slowly; though if anything this demonstrates the challenge.

It’s also a harder problem to solve than Google’s algorithm.

After two decades Microsoft seems to have brought in support, but they left the hard part to others.

They seem to be very dominant on the problem of “here’s my dataset, please provide me with an api / interface to search it”

When searching for something on HN, I get much better results using Google and site: rather than using the embedded algolia search.

I would have to think that this would be true of a corporate data set as well.

Their api might be what sets them apart, though. I’m not sure… not in that space.

Can you elaborate?

If someone searches for something on one engine and it doesn’t have the results the “meta” search engine could be checked and direct the user to another engine that would better fit their query.