Utter layman's guess: While indexing has gotten easier, the web has expanded exponentially in the meantime - likely far outstripping any technical gains. I would be surprised if indexing the modern web is feasible without significant time and capital.

Also, I can think of a couple specialized engines that do exist: Google Scholar and Shodan. There are probably more I'm unfamiliar with.

Not only is it infeasible, but there's so much garbage out there that we have no simple way to filter it out from scratch to the point that anyone could actually use it. Google and Bing have feedback loops with their users that prevent crap from rising to the top of search results. A plain index (or even using something like pagerank) wouldn't have this huge benefit, and you'd never be able to get your search engine off the ground.

The problem is that Google and Bing really are duopolistic.

You'd think that a monopoly could just break instantly if all it required was typing in a different URL.

But modern search engines are reliant on machine learning on mind-bogglingly enormous troves of real human interaction data.

If you truly outsmart Google by inventing a better mousetrap, it's worth fuck-all. Your solution will probably require more usage data than you will ever be able to collect, because nobody will use your search engine while it still produces poor results.

This is the conventional wisdom. On the plus side, it means that those of us who want to build innovative search engines don't have much competition!

Well, ML on 100 billion searches is probably going to more closely approximate user intent than a living brain in a tank, because that brain doesn't know what the hell "lkw attachment" means.

Looks like german-english bilingual logistics professionals are looking for truck parts vendors, while teenage Americans are looking for hidden locations of laser focusing and enhancement devices within the video game Wolfenstein The New Order.

Do you envision a second route to that answer?

Well, the interface offered by blekko's Izik tablet search engine was that it would show 2 categories in the answer, one related to automotive and one related to games.

Google has done this for a while, but at the level of recognized entities, not at the level of individual queries. Is that how the engine you're referring to worked too? Eg, would the "lkw attachment" example above have partitioned results?

In Izik's tablet interface, each category was a separate row of results. So in this example, there would be 2 rows, one for automotive parts and one for games, and if you scroll horizontally in a row you get more results in that category.

I think that's what you meant by partitioned results.

Google computes this internally but I've never seen them use it for anything other than having diversity in their top 10 results.

Yea it's explicitly separated, just not shown at the same time. For example, if you search for "kings", there will be a couple of bubbles at the top of the page with different entities: "Kings" (2017 film), "Sacramento Kings" (baseball team), etc. Clicking on one of those will show you a list of results that only pertains to that entity. This feature has been around for years, and is part of the series of "things, not strings" features they've been working on.

As I said, Google is pretty conservative about this and other entity-based features, so they definitely wouldn't do it got something like "lkw attachment". My question was whether Izik triggered this feature in such cases or not.

That Google feature is like "related queries", that kind of feature has been around for more than a decade. If you click on the "Kings (2017 Film)" link it runs a search for [Kings 2017]... which just adds 2017 as a keyword to a conventional search. No semantic search is involved.

Izik would show you film-related website results for the film category.

State Monopolies do a good job.

Setup grants for a domestic internet fund to incubate tech, ban competitors and blamo!

Works miracles in China.

The same underpinning is in many Francophone nations.

Not sure you'll find many products banned in francophone countries.

It is true, however that many companies don't bother to localize.

More so that they do have legal frameworks to restrict and ban English speech. They could easily force it online, but oddly enough don't.

It seems they are content with raiding the Canadian film board.

I think these exist, they've just become more specialized. Think of these sites you might have seen before:

- alternativeto.net and similar - Google Scholar / Semantic Scholar - Every sandboxed social network (Facebook, Twitter, Tumblr) - A variety of Instagram searching sites - Alternative App stores for Android

And there's room for plenty more sites like this.

Attempting to return a good response for anything in the search box is a bottomless problem of unclear utility. Being more focused makes the work easier and the value for the user clearer.

Google jumped into the market when it was still possible to compete. And they came to the table with a product that was faster, better, cheaper, and more easy to monetize. Google-style data centers reduced costs, sharding and map-reduce plus streamlined design improved speed, pagerank improved quality, low cost fast searches meant that low cost advertisements could still bring in a lot of RoI. From that kernel they grew to dominance, becoming synonymous with the very term "search". Now we live in a different era, one where search is integrated in everything on every platform and where replacing the default search engine is a huge uphill battle.

Let's say that someone creates a better search engine, how would people actually use it? I'll tell you how most people who bothered to use it would integrate into their routines. Firstly they would still use google for everything day to day. They would still use google maps for directions. They would still use gmail for mail. They would still use google search as the semantic front-end for their browsing. Only after they performed a search using google that produced unsatisfactory results would they then pull out the better search engine and make use of it for that one isolated search. And that's the problem, because that scenario is very hard for the better search engine maker to monetize while google would continue reaping the major monitization haul for the vast majority of search uses for that user. And going from zero to completely integrating into a user's experience in the same way that google does now is not a realistic prospect for most startups.

I'd love a search engine with more querying power limited to a niche. But I doubt limiting topical scope would do much to keep computational complexity from blowing up. A while back some startup was charging per search to work around that.

blekko provided 2,000 niches and it wasn't enough to keep us alive. Also, you need a web-wide crawl to build a good niche engine.

I wasn't really thinking a niche topic but a niche use-case. Google is bad at doing literal, verbatim string searches these days and all I want sometimes is like Google 2005ish era search, just PageRank based and no modern machine learning, etc. You can have it simpler than Google did at the time since SEO against your niche engine is unlikely, so tech wise you might be more like 1998 Google. The main bottleneck seems to be what others here are suggesting: there's just too much web to index these days unless you're Bing-scale or bigger.

> Google is bad at doing literal, verbatim string searches these days and all I want sometimes is like Google 2005ish era search, just PageRank based and no modern machine learning, etc

You can get most of the way to this with the verbatim option, but I think they make it difficult to make the default.

> By using the Verbatim tool Google will not make the following changes: • Personalizing your search using websites you have visited before; • Including synonyms of your search terms; • Automatic spelling corrections; • Searching for words with the same stem e.g. “Shopping” when searched for “shop”; • Finding results that match similar terms to those in your query.

Well that's sort of what I'm thinking of too. I'm talking about the idea that this might be doable by limiting topical scope. So for example you'd only index some portion of tech related sites.

I think the transformation has been more subtle, App Stores SEO is a thing, and voice searches are becoming more common. Document searches is still Google though.

Because Google and Bing are in the advertising business, not the search engine business. (There is no "search engine industry".)

Their goal is ad clickthrough, not accurate search results.

Sort of. They need good search results so users return and have a probability of ad click again.

I think it is hard to get as good as google without spending tons of money and/or time and then get the market share to start possibly make any money.

for me the image search by google is broken and I can't find useful stuff with it