News for hackers. Stuff that matters.
Re:Ask HN: Is there room for another search engine? by thesmallestcat on Sunday March 26, 2017 @01:46PM (#13961975)
There are way more than two things that Google does wrong. Remapping my search terms into oblivion so it can pretend it's fast is the worst one. Especially when this happens to a query I've modified to quote "every" "single" "flipping" "term." I think Google is cheating, and that their usable index is much shallower than they'd have you believe.

What's needed is a search engine with functional queries (as opposed to Google, which now only operates in "the user is drunk" mode), that doesn't give a damn about your robots.txt, and that can capture content in a way that is more akin to archive.org than Google's shoddy and increasingly absent cache.

Another issue is spam/false matches. Why does Google return illegitimate results? Because, let me tell you, any search for "some nifty computer book pdf" returns pages upon pages of bogus links leading to ad link mazes. A crawler should be able to trivially crawl such a page, determine that no PDF is linked, and blacklist the result, but this doesn't happen.

Google is slow and preoccupied. Their business is ripe for disruption.

Re:Ask HN: Is there room for another search engine? by cocktailpeanuts on Sunday March 26, 2017 @02:04PM (#13962112)
Those are all things Google can already do but decide not to. Which means if any startup attempts and gets a moderate success, google will of course decide to do the same. Your startup will be nothing more than an R&D department Google uses for free.

Disruption happens by doing something completely different that happens to cannibalize the incumbent's business in a way that they can't change. All these little features mentioned are something Google can easily do so these are not really disruptive.

Re:Ask HN: Is there room for another search engine? by thesmallestcat on Sunday March 26, 2017 @02:12PM (#13962151)
Keep in mind that it's been 20 years since Google's differantiator (PageRank) was invented. It's been about 10 years since Google's last meaningful changes to its search UI (autocomplete and "instant" search). So, while I respect Google's omniscience and omnipotence, I disagree strongly that Google/Alphabet are in a position to respond to a serious upstart competitor. Their primary response, as you hinted would be to acquire/acquihire, which is unimaginative and not exactly how you remain on the cutting edge. That strategy falls on its face as soon as it meets that one damn company that doesn't want to be purchased.
Re:Ask HN: Is there room for another search engine? by cocktailpeanuts on Sunday March 26, 2017 @02:25PM (#13962234)
There are already many "serious upstarts" that are doing just fine but will never reach the status of Google, like duckduckgo, bing, etc.

Some even say bing has better search quality than google. But that's not what matters. No matter how good the product is, if you can't get distribution it doesn't matter. Also, even if you do have distribution, it may not be the right context so it won't be effective.

If anyone is trying to really build a search engine to compete directly against Google, I can only say good luck, and prepare to be the most patient person on earth, because it won't happen fast, and probably won't even happen. Instead, while you're slowly making progress day by day with small traffic increase, you'll see some random new thing come out that had no intention of becoming a "search engine" (just like Youtube) and "disrupt" Google.

I have seen a lot of people talk about "disrupting something", and also have seen a lot of people who actually have disrupted their industry. The former are just wannabes, and the latter never said anything about "disrupting" something. They just saw an opportunity and went for it. And it ended up disrupting. Hope this makes sense.

p.s. Google doesn't operate on the original pagerank anymore, there are tons of other things going on underneath. Also they now own many innovative AND popular "search UI" both through acquisitions and their R&D. Google Maps owns a lot of the map space, Youtube owns the video search space, and so on. Look deeper and you'll find even more that you've been taking for granted.

Re:Ask HN: Is there room for another search engine? by thesmallestcat on Sunday March 26, 2017 @02:37PM (#13962307)
We can agree to disagree about Google's immortality, but I take issue with you treating DDG and Bing as serious challengers to Google. Obviously Microsoft has no chance, they're worse off than Google re: ossification and Bing is a hedge at best. DDG is one step above a lifestyle business and only innovates at the UI level, even their efforts there are forgettable. Their major differentiator (data privacy) is meaningless to almost all users, and ironically torpedoes many interesting monetization strategies that are not ad-focused, as well as research initiatives.

Baidu and Yandex are serious but are comfortable in their respective positions. That said, I wouldn't be surprised to see a Google challenger spring up from the other side of the world, where Google's position may not be seen as so unassailable.

Re:Ask HN: Is there room for another search engine? by greglindahl on Sunday March 26, 2017 @03:15PM (#13962545)
While google has a much stronger brand than bing, they give roughly equivalent results except for a few categories such as "paste the error message I just got into the search box" queries. Google's much better for those longer queries. HN readers are probably more likely than average to make them, too.
Re:Ask HN: Is there room for another search engine? by cassowary on Monday March 27, 2017 @01:53AM (#13965462)
Hm yeah. I use DDG except for error messages when I use Google.
Re:Ask HN: Is there room for another search engine? by cocktailpeanuts on Sunday March 26, 2017 @03:12PM (#13962520)
> We can agree to disagree about Google's immortality

Dude where did I say Google is immortal? I'm just saying it's a wrong approach to try to kill google this way. There are many good ways to do it and coming from "How do i disrupt Google" is the worst possible way.

> Obviously Microsoft has no chance, they're worse off than Google re: ossification and Bing is a hedge at best.

Nothing is obvious when we talk about disruption. Why does Microsoft have no chance? I am pretty sure they keep Bing alive because they are looking at the future where they finally gain control over the platform via VR/AR/or whatever, and then use Bing to power the experience on those platforms, which actually may be the case Google can't compete against. But this is Microsoft. If you're a small startup, you shouldn't be doing this.

> DDG is one step above a lifestyle business and only innovates at the UI level, even their efforts there are forgettable. Their major differentiator (data privacy) is meaningless to almost all users, and ironically torpedoes many interesting monetization strategies that are not ad-focused, as well as research initiatives.

Privacy is a great feature, especially in 2017, and I think DDG has the highest chance of getting any market share with this "let's make a search engine to disrupt google" approach (but again, this likely won't happen and is a foolish approach. It will stay down there as an "alternative" until the founder runs out of steam, OR if there's a huge shift in how people experience the Internet and somehow takes off completely independent from the company's effort)

> Baidu and Yandex are serious but are comfortable in their respective positions.

Baidu and Yandex will never work out side of their countries because of the cultural barrier. In fact it's not just those, every country with significant Internet penetration rate has their own dominant search engine. In those countries Google is nothing. Which means this cuts both ways. So No. Baidu and Yandex are NOT serious contenders just like Google is not a serious contender in Russia and China.

Re:Ask HN: Is there room for another search engine? by bshimmin on Sunday March 26, 2017 @04:04PM (#13962903)
I wonder, is DuckDuckGo's name a problem for adoption? I have no idea what it's supposed to mean, but it sounds very silly and I don't much enjoy recommending it to people in real life, at least in part for that reason. ("Google" was a silly name too, but at least it was just one syllable of silliness.)

Privacy is probably slowly growing into a big deal in the minds of regular people, but when the incumbent works, and works better (in my experience Google does just return better results than DDG - and I really did give it a try, in earnest, for a solid week a few months back...), privacy alone isn't going to be enough at this point.

Re:Ask HN: Is there room for another search engine? by cocktailpeanuts on Sunday March 26, 2017 @04:28PM (#13963079)
Yeah like you pointed out, if your product stands out so much that nothing else matters, then you can name it whatever you want. But duckduckgo is just a glorified meta search engine and they are not doing anything completely unique. In this case branding matters and I think their name is really bad for branding. There are many examples of really bad naming, like cockroachdb and swagger. Swagger ended up changing their name to OpenAPI (Thank god) but I don't know what the cockroachdb guys are thinking. I don't want to use a DB named after a bug (pun intended).
Re:Ask HN: Is there room for another search engine? by corobo on Monday March 27, 2017 @01:50AM (#13965452)
> I wonder, is DuckDuckGo's name a problem for adoption

For me it is too, yeah. They even own duck.co which would be a better brand. It'd be a pain in the arse to tell others to use to start ("duck dot com?" "no, dot co") but nowhere near as silly sounding as duck duck go.

Re:Ask HN: Is there room for another search engine? by flukus on Monday March 27, 2017 @08:17PM (#13972933)
> Privacy is probably slowly growing into a big deal in the minds of regular people, but when the incumbent works, and works better (in my experience Google does just return better results than DDG - and I really did give it a try, in earnest, for a solid week a few months back...), privacy alone isn't going to be enough at this point.

I've had the opposite experience lately, DDG has been almost identical to the google results, to the point where the seem to be calibrating there own searches with google. Unfortunately it means it also suffers the same problems.

Re:Ask HN: Is there room for another search engine? by jaredsohn on Sunday March 26, 2017 @04:06PM (#13962913)
>I have no idea what it's supposed to mean

I've understood it to be a reference to https://en.wikipedia.org/wiki/Duck,_duck,_goose; the wikipedia article on duckduckgo confirms.

Re:Ask HN: Is there room for another search engine? by bshimmin on Sunday March 26, 2017 @04:21PM (#13963020)
Is this an American thing? I have never heard of it.
Re:Ask HN: Is there room for another search engine? by cassowary on Monday March 27, 2017 @02:16AM (#13965547)
I played Duck Duck Goose when I was a kid in Australia.
Re:Ask HN: Is there room for another search engine? by gregjw on Monday March 27, 2017 @06:06AM (#13966361)
I played Duck Duck Goose when I was a kid in the United Kingdom.
Re:Ask HN: Is there room for another search engine? by amyjess on Sunday March 26, 2017 @05:12PM (#13963325)
I'm convinced that the only reason Microsoft is keeping Bing alive is to keep Windows 10 users from funneling money to Google by default.
Re:Ask HN: Is there room for another search engine? by nojvek on Monday March 27, 2017 @12:45AM (#13965239)
Windows use is pretty large and Bing is profitable. Edge is funded by bing profits I believe.

Only large companies can afford to make browsers. They are insanely complicated beasts now. Google because ad and Android, Microsoft because windows and Bing, Apple because of ios. Firefox is facing struggles and Opera gave up to webkit.

Disrupting Google search is going to be very hard. Google invests more than anyone in AI. If someone were to disrupt Google, they have to have very strong models of the world, natural language parsing, image recognition, knowledge reasoning, troves of data and an ocean of servers and fibre.

Google hasn't been sitting idle, they've been disrupting themselves and pushing towards those fronts.

Re:Ask HN: Is there room for another search engine? by Animats on Sunday March 26, 2017 @03:10PM (#13962508)
prepare to be the most patient person on earth, because it won't happen fast, and probably won't even happen

Maybe not. The Cuil experience is significant. The PR for the launch was good, there was a huge initial traffic spike, and then users discovered the product sucked and traffic dropped. The system at launch was terrible. Six months later it wasn't so bad, but nobody cared by then. They launched too early. But they did, briefly, have mindshare, lost through technical ineptitude.

Re:Ask HN: Is there room for another search engine? by jacquesm on Sunday March 26, 2017 @04:52PM (#13963217)
I think the bigger problem with Cuil is they over-hyped their offering before launch which could lead to nothing but disappointment.
Re:Ask HN: Is there room for another search engine? by moultano on Sunday March 26, 2017 @04:42PM (#13963157)
I'd argue that showing answers on the search result page is a bigger innovation than "instant" was. It also requires much more new technology.
Re:Ask HN: Is there room for another search engine? by 19eightyfour on Sunday March 26, 2017 @02:58PM (#13962444)
They could erase competitors from results.
Re:Ask HN: Is there room for another search engine? by kjhughes on Sunday March 26, 2017 @01:55PM (#13962048)
To disable remapping of terms in Google, use verbatim search: On the search results page, choose "Search tools -> All results -> Verbatim"
Re:Ask HN: Is there room for another search engine? by dredmorbius on Sunday March 26, 2017 @03:42PM (#13962735)
There's no syntax of which I'm aware which tells Google to not "correct" or expand the results.

When I'm seeking specific matches, I'm seeking really fucking specific matches. Google annoy me to no end.

Re:Ask HN: Is there room for another search engine? by gamzer on Monday March 27, 2017 @07:07AM (#13966658)
I believe it was possible in the past using URL parameters (e.g. complete=0 to disable auto-suggest) but it does not seem to work anymore.

Here it was documented in 2012:

https://airstraining.wordpress.com/2012/06/13/google-search-...

There are (or at least were) others with related effects.

Re:Ask HN: Is there room for another search engine? by mtreis86 on Sunday March 26, 2017 @05:25PM (#13963385)
Boolean search would be wonderful
Re:Ask HN: Is there room for another search engine? by user5994461 on Sunday March 26, 2017 @05:21PM (#13963372)
+word

It used to force to search the word exactly as written.

Re:Ask HN: Is there room for another search engine? by dredmorbius on Sunday March 26, 2017 @07:53PM (#13964106)
"Used to" is the operative phrase. That was among the negative consequences of Google+, as +<term> notation apparently was going to be reserved or repurposed for that somehow, but never was.

I've done some large-scale searching where the most relevant detail is how many results are returned, most particularly for a specific domain. (For which, incidentally, there's no handy mechanism to accomplish, so it's <array of terms> * <array of domains>, and multiplicative explosion of searches, plus about a 45s timeout per query to avoid triggering bot defenses by Google.

Such as this:

https://www.reddit.com/r/dredmorbius/comments/3hp41w/trackin...

Re:Ask HN: Is there room for another search engine? by donretag on Sunday March 26, 2017 @06:04PM (#13963572)
The plus/+ notation has been removed. You must know surround the term in quotes "word".
Re:Ask HN: Is there room for another search engine? by wildpeaks on Monday March 27, 2017 @11:12AM (#13968338)
I wish there was a way to use both Verbatim and Date Range at the same time: you end up having to choose between recent but irrelevant results, or relevant but obsolete results.
Re:Ask HN: Is there room for another search engine? by icebraining on Sunday March 26, 2017 @02:14PM (#13962167)
Another issue is spam/false matches. Why does Google return illegitimate results? Because, let me tell you, any search for "some nifty computer book pdf" returns pages upon pages of bogus links leading to ad link mazes. A crawler should be able to trivially crawl such a page, determine that no PDF is linked, and blacklist the result, but this doesn't happen.

The problem is interesting, but I think you make it seem easier than it is. What if there's a CAPTCHA, as there often is? Should the engine still ignore it? That could lead to a whole lot of missed content.

Re:Ask HN: Is there room for another search engine? by thesmallestcat on Sunday March 26, 2017 @02:20PM (#13962201)
I'd argue that paywalled sites, Captcha-guarded pages, things like that are second-class content and should be treated as such. If somebody wants to dig deeper and see such results, that should be an option, but as you said, this is a tough problem to solve satisfactorily and the truth lies somewhere inbetween. An obstacle to this happening right now is that Google doesn't want to shake the advertising tree too hard. Their business model is spammy so they tolerate spamminess. That's the hardest problem of all, and one I don't think anybody's close to solving yet: How do you monetize web search without automatically wrapping results in ads? But some bright person will come up with an answer and hopefully defeat this huge conflict of interest.
Re:Ask HN: Is there room for another search engine? by bambax on Sunday March 26, 2017 @04:05PM (#13962912)
> a query I've modified to quote "every" "single" "flipping" "term."

Use Tools > Verbatim, it's much faster than quoting every search term, and it works better.

Re:Ask HN: Is there room for another search engine? by Analemma_ on Sunday March 26, 2017 @01:57PM (#13962064)
Any search crawler that ignores robots.txt is going to be blocked by site operators in a hurry.
Re:Ask HN: Is there room for another search engine? by thesmallestcat on Sunday March 26, 2017 @01:59PM (#13962081)
If Google today started ignoring robots.txt, not many people would start blocking Google's crawlers, assuming they continued to do their job efficiently. robots.txt is security by obscurity at best.
Re:Ask HN: Is there room for another search engine? by Analemma_ on Sunday March 26, 2017 @02:17PM (#13962185)
People wouldn't block Google because allowing Google scraping offers a return in the form of more traffic that offsets the cost. $BRAND_NEW_ENGINE wouldn't have that advantage.
Re:Ask HN: Is there room for another search engine? by thesmallestcat on Sunday March 26, 2017 @02:23PM (#13962224)
Google's crawlers don't exactly perform 2FA before they crawl a website. If impersonating Google's crawlers doesn't suit your fancy, there are all manner of ways to anonymize a crawler. So I think blocking is one of the least interesting challenges. The other side of the coin is robots.txt is not inherently adversarial, and a crawler could waste quite some time and energy crawling truly meaningless content. That, in my mind, is the interesting challenge: Obsoleting robots.txt with an intelligent (and gentle) crawler.
Re:Ask HN: Is there room for another search engine? by desdiv on Sunday March 26, 2017 @03:03PM (#13962475)
Google doesn't feel like being impersonated so they offer an easy way to authenticate Googlebot:

https://support.google.com/webmasters/answer/80553?hl=en

Re:Ask HN: Is there room for another search engine? by angry-hacker on Monday March 27, 2017 @06:06AM (#13966365)
They do ignore robots.txt sometimes, the pages just font end up in serps.
Re:Ask HN: Is there room for another search engine? by user5994461 on Sunday March 26, 2017 @05:22PM (#13963375)
And more importantly: It's illegal :D