What's needed is a search engine with functional queries (as opposed to Google, which now only operates in "the user is drunk" mode), that doesn't give a damn about your robots.txt, and that can capture content in a way that is more akin to archive.org than Google's shoddy and increasingly absent cache.

Another issue is spam/false matches. Why does Google return illegitimate results? Because, let me tell you, any search for "some nifty computer book pdf" returns pages upon pages of bogus links leading to ad link mazes. A crawler should be able to trivially crawl such a page, determine that no PDF is linked, and blacklist the result, but this doesn't happen.

Google is slow and preoccupied. Their business is ripe for disruption.

Disruption happens by doing something completely different that happens to cannibalize the incumbent's business in a way that they can't change. All these little features mentioned are something Google can easily do so these are not really disruptive.

Some even say bing has better search quality than google. But that's not what matters. No matter how good the product is, if you can't get distribution it doesn't matter. Also, even if you do have distribution, it may not be the right context so it won't be effective.

If anyone is trying to really build a search engine to compete directly against Google, I can only say good luck, and prepare to be the most patient person on earth, because it won't happen fast, and probably won't even happen. Instead, while you're slowly making progress day by day with small traffic increase, you'll see some random new thing come out that had no intention of becoming a "search engine" (just like Youtube) and "disrupt" Google.

I have seen a lot of people talk about "disrupting something", and also have seen a lot of people who actually have disrupted their industry. The former are just wannabes, and the latter never said anything about "disrupting" something. They just saw an opportunity and went for it. And it ended up disrupting. Hope this makes sense.

p.s. Google doesn't operate on the original pagerank anymore, there are tons of other things going on underneath. Also they now own many innovative AND popular "search UI" both through acquisitions and their R&D. Google Maps owns a lot of the map space, Youtube owns the video search space, and so on. Look deeper and you'll find even more that you've been taking for granted.

Baidu and Yandex are serious but are comfortable in their respective positions. That said, I wouldn't be surprised to see a Google challenger spring up from the other side of the world, where Google's position may not be seen as so unassailable.

Dude where did I say Google is immortal? I'm just saying it's a wrong approach to try to kill google this way. There are many good ways to do it and coming from "How do i disrupt Google" is the worst possible way.

> Obviously Microsoft has no chance, they're worse off than Google re: ossification and Bing is a hedge at best.

Nothing is obvious when we talk about disruption. Why does Microsoft have no chance? I am pretty sure they keep Bing alive because they are looking at the future where they finally gain control over the platform via VR/AR/or whatever, and then use Bing to power the experience on those platforms, which actually may be the case Google can't compete against. But this is Microsoft. If you're a small startup, you shouldn't be doing this.

> DDG is one step above a lifestyle business and only innovates at the UI level, even their efforts there are forgettable. Their major differentiator (data privacy) is meaningless to almost all users, and ironically torpedoes many interesting monetization strategies that are not ad-focused, as well as research initiatives.

Privacy is a great feature, especially in 2017, and I think DDG has the highest chance of getting any market share with this "let's make a search engine to disrupt google" approach (but again, this likely won't happen and is a foolish approach. It will stay down there as an "alternative" until the founder runs out of steam, OR if there's a huge shift in how people experience the Internet and somehow takes off completely independent from the company's effort)

> Baidu and Yandex are serious but are comfortable in their respective positions.

Baidu and Yandex will never work out side of their countries because of the cultural barrier. In fact it's not just those, every country with significant Internet penetration rate has their own dominant search engine. In those countries Google is nothing. Which means this cuts both ways. So No. Baidu and Yandex are NOT serious contenders just like Google is not a serious contender in Russia and China.

Privacy is probably slowly growing into a big deal in the minds of regular people, but when the incumbent works, and works better (in my experience Google does just return better results than DDG - and I really did give it a try, in earnest, for a solid week a few months back...), privacy alone isn't going to be enough at this point.

For me it is too, yeah. They even own duck.co which would be a better brand. It'd be a pain in the arse to tell others to use to start ("duck dot com?" "no, dot co") but nowhere near as silly sounding as duck duck go.

I've had the opposite experience lately, DDG has been almost identical to the google results, to the point where the seem to be calibrating there own searches with google. Unfortunately it means it also suffers the same problems.

I've understood it to be a reference to https://en.wikipedia.org/wiki/Duck,_duck,_goose; the wikipedia article on duckduckgo confirms.

Only large companies can afford to make browsers. They are insanely complicated beasts now. Google because ad and Android, Microsoft because windows and Bing, Apple because of ios. Firefox is facing struggles and Opera gave up to webkit.

Disrupting Google search is going to be very hard. Google invests more than anyone in AI. If someone were to disrupt Google, they have to have very strong models of the world, natural language parsing, image recognition, knowledge reasoning, troves of data and an ocean of servers and fibre.

Google hasn't been sitting idle, they've been disrupting themselves and pushing towards those fronts.

Maybe not. The Cuil experience is significant. The PR for the launch was good, there was a huge initial traffic spike, and then users discovered the product sucked and traffic dropped. The system at launch was terrible. Six months later it wasn't so bad, but nobody cared by then. They launched too early. But they did, briefly, have mindshare, lost through technical ineptitude.

When I'm seeking specific matches, I'm seeking really fucking specific matches. Google annoy me to no end.

Here it was documented in 2012:

https://airstraining.wordpress.com/2012/06/13/google-search-...

There are (or at least were) others with related effects.

It used to force to search the word exactly as written.

I've done some large-scale searching where the most relevant detail is how many results are returned, most particularly for a specific domain. (For which, incidentally, there's no handy mechanism to accomplish, so it's <array of terms> * <array of domains>, and multiplicative explosion of searches, plus about a 45s timeout per query to avoid triggering bot defenses by Google.

Such as this:

https://www.reddit.com/r/dredmorbius/comments/3hp41w/trackin...

The problem is interesting, but I think you make it seem easier than it is. What if there's a CAPTCHA, as there often is? Should the engine still ignore it? That could lead to a whole lot of missed content.

Use Tools > Verbatim, it's much faster than quoting every search term, and it works better.

https://support.google.com/webmasters/answer/80553?hl=en

My experience with Google is the opposite of this. When people are searching google they are looking for information. When people go to Amazon they have their credit card in their hand.

Most of my google-ad clicks (85% last check) are clearly not interested in buying what they are looking for. They are only on my page for seconds. If I try to qualify my ads better I lose 'quality score,' which is a measure that is entirely, at least to a first-order approximation, about whether or not people will click the ad. That, if my ad say 'This site is X' I get a good quality score but shitty leads. If I say 'This site is X for $49.99' I get shitty quality score but the people who come to the site (the clicks I have to pay for) are ready to buy.

The profitability isn't because their initial hypothesis of Search Marketing (that people are searching what they want to purchase and will therefore purchase) was correct, but because they are far-and-away the search winners and if you want to be found at all you have to play it.

At least, in my experience and IMHO.

And – as just another anecdote – I have had continuing success over the last 8 to 10 years with search ads, while not having a single conversion with anything else (Facebook, youtube, twitter etc.) as far as I can tell.

Also, it seems to me that both weaknesses described, legitimacy and provenance, are actually good for Gs core ad business, because if legitimate advertisers could reliably rank highly without paying, they'd have no interest to pay to out rank the competition. The competition who are buoyed by the nebulous practices Google pretends to frown upon, but for the sake of a delicate balance cannot justify completely stamping out.

The perhaps unpalatable and unutterable truth is that ad-supported search is a delicate business: be good enough to attract eyeballs, but not so good that advertisers who could pay you have no need to pay.

ActionCookBook accurately recreates my experience. https://twitter.com/actioncookbook/status/834439563032555521

   ME: [views product online] Hmm. Nah.
   [changes website]
   PRODUCT: we meet again
   ME: Sorry, no [changes again]
   PRODUCT: bitch this ain't over
   ME: FINE. FINE. I will buy this rug. Just leave me at peace.
   REST OF INTERNET: this dude loves rugs, let's get him, boys

> Google is also bad at provenance - figuring out that site A is using text derived from site B, and thus B should be ranked higher.

The part I dispute is the last part. What matters is the user perspective. So of course a site that does nothing but scrape another one should rank lower, but many scrapers add value, if none other than UI-wise. So it's not obvious that the site of origin should rank higher.

(Also, how about DuckDuckGo? I've got the (admittedly gut) feeling that it should at least outperform Ask and Excite.)

[1] https://www.netmarketshare.com/search-engine-market-share.as... [2] https://www.searchenginejournal.com/august-2016-search-marke...

https://en.wikipedia.org/wiki/DuckDuckGo

And Bing, Yandex, and Yahoo have active crawlers.

Yeah, I don't think many people will care about the difference between good and perfect. You might be able to find a niche in search that Google is ignoring, but you would have a hard time expanding from there into general search.

In that sense Google is a bet against technology - you would invest in Google if you believe the Web's going to stay the same for a long time and nothing will replace it.

Without needing humans to tell them what the results mean.

So no, this will not be the next direction - unless there is a universal model of machines that understand things. But there isn't, and there won't be for at least another 10 years.

IMHO, the other pieces of the picture, and what i see happening, is search engines or a service, will index entities and relationships ( in the ontology sense ), and return results enriched with that data through something like an extended microdata vocabulary. There will be an API where machine consumers can query entities and relationships, and see those attributed to their sources in the webpage. This prediction is nothing new and has been consistently foretold by ai and information retrieval augurs for decades. The difference is that I see this moving beyond the realm of expert systems in large corporations and into the realm of being an API generally accessible to anyone.

I think it's very possible that, as you say, a current incumbent search engine may not consider the provision of such a service its job. Which makes things very interesting for potential new entrants in the semantic search market.

Other business types could surely supply these entities and relationships however who better to be involved in the supply chain than search engines because they hold a hose containing nearly all the world's information, and in the other hand a market hungry for all the world's information.

But this type of service could be niche because most humans are not going to care about getting ontology data in their search results. So even though parsing out entities and relationships could improve the usefulness and accessibility of information there might not be enough universal demand for someone like Google to really care about it.

With these caveats, I think the trend from information, to knowledge, is a very natural and already apparent progression for search engines. The type of API described here is possible today given the right incentives.

Edit: There should be a differentiator for new search engine, there is more room (problems to solve) in Q&A search and Discovery.

Google's clean interface (basically just a search box) was so revolutionary that it's hard to comprehend today - I still remember it took me a few days to even take it serious, it was so thin and lightweight. PageRank was a gigantic step forward in SERP relevancy and even though it was gamed eventually by link farms and paid backlinks, it took a lot of time and effort and the first few years saw few spammers.

Well, yes, it was simply that. It was the difference between a search engine that works and one that doesn't. For example, most sites back then had tons of text with the same color of the background, with words that would get you an higher rank on the previous generation of search engines. Also, the search engines had very heavy pages, mostly with news, and they were known as "portals". Google was a blast in comparison.

Before Google, the dominant search engine was AltaVista, which was awful. The UI was so crowded that you had to search for the search bar, and it was so full of graphics that if you were on dialup, loading took forever unless you turned images off.

And, yeah, PageRank helped. It wasn't uncommon for you to find what you were looking for on page 3 or 4. Google had what you were looking for on page 1, always. In fact, pre-Google search was so bad that if you were looking for anything obscure, you had to use multiple search engines. When I'd get serious about finding something, I'd fire up AltaVista, Infoseek (Disney/ABC/Go.com), Hotbot, WebCrawler (AOL), etc. in separate browser windows and search on all of them. And for the lazy, there was MetaCrawler, which automated this; I didn't use it often, mostly because I kept forgetting about it, but when I remembered to use it, it was a godsend.

http://royal.pingdom.com/2008/09/05/googles-first-website-fr...

At the time, their main competitors were already expanding into other markets, and Google was refreshing for doing one thing and doing it well, compared to its competition. It didn't clutter, it didn't try to distract with other services that they offered. It simply gave you the results you queried as a plain list of links, and got out of your way so you could browse those links at your leisure.

At that point Google had a smaller index, which hurt their results, but I thought that was better than getting porn results while at work.

If so, why is Yahoo Slurp still showing up in my apache logs?

https://help.yahoo.com/kb/SLN22600.html