News for hackers. Stuff that matters.
Re:Building a Search Engine from Scratch by bla3 on Friday December 06, 2019 @02:57PM (#21725396)
That Cliqz is trying to actually build a new search stack is commendable. This is way more exciting to me than DuckDuckGo and other services that just package up Bing search results under different branding.

I'm skeptical that they'll be successful, but I wish them the best. They should market (and engineer) strongly on privacy since that's where Google is weak.

Re:Building a Search Engine from Scratch by mrb on Friday December 06, 2019 @03:56PM (#21725906)
«DuckDuckGo and other services that just package up Bing search results»

Had to look that up. I found https://help.duckduckgo.com/results/sources/ Bing is just one of "hundreds of vertical sources delivering" results to DuckDuckGo.

Re:Building a Search Engine from Scratch by bla3 on Friday December 06, 2019 @07:43PM (#21727625)
That page basically confirms this. It says

> To do that, we've developed an open source Instant Answer platform called DuckDuckHack,

which links to https://duckduckhack.com/ which says "DuckDuckHack is now in Maintenance Mode".

And the "four hundred sources" link links to 400 special case replies. They are probably useful, but fire rarely. It's basically Bing, and that page is a bunch of spin.

Re:Building a Search Engine from Scratch by mrb on Friday December 06, 2019 @09:06PM (#21727968)
Interesting, thanks.
Re:Building a Search Engine from Scratch by mda on Friday December 06, 2019 @04:00PM (#21725932)
Bing is the primary source though.
Re:Building a Search Engine from Scratch by gregschlom on Friday December 06, 2019 @07:16PM (#21727466)
I'm not sure that's true. If you search the same query on both Bing and DDG, you'll see pretty different results.
Re:Building a Search Engine from Scratch by manojlds on Saturday December 07, 2019 @12:10AM (#21728603)
I played with Yahoo BOSS a lot as a undergrad and I could tweak it to get "better" results than Yahoo Search for certain queries. That Bing and DDG have different results doesn't really say anything.
Re:Building a Search Engine from Scratch by lisper on Saturday December 07, 2019 @02:54AM (#21729063)
How does ddg do that without running afoul of bing's ToS?
Re:Building a Search Engine from Scratch by detaro on Saturday December 07, 2019 @09:58AM (#21730567)
Presumably they're using Bing's API directly, and have a suitable legal agreement with Microsoft in place.
Re:Building a Search Engine from Scratch by lisper on Saturday December 07, 2019 @12:32PM (#21731439)
I don't see how such a deal could possibly work. Microsoft's entire history and culture has been devoted to squashing nascent competitors whenever possible. So why in the world would MS license Bing to DDG? If DDG can monetize that traffic more effectively than Bing by not tracking (which seems unlikely) then why does MS just stop tracking on Bing? And if they can't, then how would they afford to pay the license fee, which surely must be at least as high as the lost revenue from the diverted traffic?
Re:Building a Search Engine from Scratch by detaro on Saturday December 07, 2019 @01:19PM (#21731694)
I imagine few people using DDG would use Bing instead, so there is little income loss from people moving Bing->DDG. If the choice is "provide DDG with a search API, get increased reach for your ad-program through an audience that otherwise wouldn't touch bing" vs "do not provide DDG with a search API, either a competitor does or DDG is way worse and everyone uses Google", why is the latter the better choice for MS?
Re:Building a Search Engine from Scratch by lisper on Saturday December 07, 2019 @01:26PM (#21731735)
Even if that were true, why would not MS just re-skin Bing as a tracker-free search engine? Or acquire DDG?
Re:Building a Search Engine from Scratch by detaro on Saturday December 07, 2019 @01:31PM (#21731765)
Make Bing tracker-free, loose (assumed) income from less precise targeting of all the current Bing users, just to maybe capture a part of DDGs market, large parts of which are going to be distrustful of Microsoft? Seems not obviously better than keeping Bing as is and being involved with DDG.
Re:Building a Search Engine from Scratch by lisper on Saturday December 07, 2019 @01:37PM (#21731800)
I didn't make myself clear. I didn't mean to get rid of Bing. MS could keep the Bing brand exactly as it is, but also present the same product under a different brand that competes directly with DDG. In other words, MS could be DDG. Never in its history has MS tolerated the existence of a competitor if there was something they could do about it, which in this case they obviously could. Why start now?
Re:Building a Search Engine from Scratch by detaro on Saturday December 07, 2019 @02:02PM (#21731924)
I'd guess a different brand attached to Microsoft would not have much more acceptance than Bing.
Re:Building a Search Engine from Scratch by lisper on Saturday December 07, 2019 @02:16PM (#21731998)
They would not have to pretend. Marketing a separate brand with its own identity is a common practice. How many people who stay at a W hotel or a Ritz Carlton or a Sheraton are aware that these brands are actually owned by Marriott? How many people who stay at a Waldorf Astoria know that it's owned by Hilton? These are not secrets. 80% of the world's economy is controlled by fewer than 1000 companies [1].

[1] https://arxiv.org/PS_cache/arxiv/pdf/1107/1107.5728v2.pdf

Re:Building a Search Engine from Scratch by iudqnolq on Saturday December 07, 2019 @03:15PM (#21732315)
And then a few days later see a HN top story that NewPrivateSearchEngine is a secret Microsoft conspiracy to destroy the world? I doubt it would go over well.
Re:Building a Search Engine from Scratch by lisper on Saturday December 07, 2019 @03:22PM (#21732347)
If you're going to indulge in that level of paranoia, how do you know that DDG is not itself a secret Microsoft conspiracy to destroy the world? If you don't know the terms under which DDG licenses Bing's search results, how do you know that those terms don't give MS complete control over DDG?
Re:Building a Search Engine from Scratch by iudqnolq on Saturday December 07, 2019 @03:31PM (#21732382)
I think I was too snide in my comment and it wasn't clear.

I meant that if Microsoft launched a privacy-focused product and hid their involvement with it they would receive extremely negative publicity on the sort of websites people who use privacy-focused products read.

Completely leaving aside the harm that would cause to the Microsoft brand, it would also be completely useless. Essentially no one would switch from DDG to Microsoft's (hypothetical) shady clone.

Your hotel examples aren't relevant because people searching for hotels and people searching for private search engines don't evaluate details about corporate ownership the same.

Re:Building a Search Engine from Scratch by lisper on Saturday December 07, 2019 @03:49PM (#21732462)
There is a difference between hiding your involvement and not going out of your way to draw attention to it.
Re:Building a Search Engine from Scratch by iudqnolq on Saturday December 07, 2019 @03:54PM (#21732478)
Yes. Do you believe that a HN discussion of a company owned by Microsoft that doesnt draw attention to that and which markets towards privacy-focused users would he received positively?
Re:Building a Search Engine from Scratch by lisper on Saturday December 07, 2019 @04:00PM (#21732517)
Do you believe that an HN discussion has the potential to move the market share needle for any MS product? More to the point, do you believe that Microsoft thinks this?
Re:Building a Search Engine from Scratch by iudqnolq on Saturday December 07, 2019 @04:06PM (#21732553)
If that product is aimed at privacy-focused users, yes.

We've gotten so far into unlikely hypotheticals I don't find this conversation interesting any more. I won't reply further in this chain. Have a nice day!

Re:Building a Search Engine from Scratch by AntiqueFig on Friday December 06, 2019 @07:08PM (#21727404)
Source?
Re:Building a Search Engine from Scratch by pheug on Friday December 06, 2019 @08:43PM (#21727881)
https://duckduckgo.com/?q=%22my+user+agent%22 - check what the snippets say.
Re:Building a Search Engine from Scratch by ksec on Saturday December 07, 2019 @09:55AM (#21730553)
So Duck Duck Go isn't actually a Search Engine, but an Interface or Proxy To Bing / or multiple search engine?

I have always thought DuckDuck were doing their own searches.

Re:Building a Search Engine from Scratch by mk89 on Saturday December 07, 2019 @05:11AM (#21729455)
If I recall, in the very beginning they were massively using Yahoo and to some extent Bing. It might be that over the years Bing has become a more reliable source.
Re:Building a Search Engine from Scratch by detaro on Saturday December 07, 2019 @09:58AM (#21730569)
Yahoo Search was based on Bing since 2009.
Re:Building a Search Engine from Scratch by mk89 on Sunday December 08, 2019 @06:56PM (#21739365)
Wow, I did not know that. Or I just forgot. Anyways,thanks for the information.
Re:Building a Search Engine from Scratch by umvi on Friday December 06, 2019 @03:22PM (#21725601)
> They should market (and engineer) strongly on privacy since that's where Google is weak.

How can you build a privacy oriented search engine and still make money?

Re:Building a Search Engine from Scratch by bla3 on Friday December 06, 2019 @03:24PM (#21725631)
By selling ads solely against the query, and figuring out a way to track conversions against generated unique referring URL instead of using cookies. Maybe it's less profitable, but it might still be profitable, and it gives you a foothold.
Re:Building a Search Engine from Scratch by MrQuincle on Saturday December 07, 2019 @05:14AM (#21729463)
Or a sponsored and unsponsored section. The only thing you pay for is for being higher up in the sponsored pane. No need for tracking. No conversion clicks tracked. Only higher on the sponsored listing.
Re:Building a Search Engine from Scratch by colejohnson66 on Saturday December 07, 2019 @12:14PM (#21731324)
Wouldn’t that prevent people with less money from buying their way up to the top? In other words, the ones with the most money (not startups, mom&pop shops, etc) would be the only ones at the top.
Re:Building a Search Engine from Scratch by jdck1326 on Friday December 06, 2019 @05:33PM (#21726679)
If one didn't mind breaking even rather than making money one could create a search engine which is free software and have groups/people host their own instances.
Re:Building a Search Engine from Scratch by rsnor on Friday December 06, 2019 @11:13PM (#21728429)
Searx (https://github.com/asciimoo/searx/wiki/Searx-instances) is exactly what you described.
Re:Building a Search Engine from Scratch by setnone on Saturday December 07, 2019 @01:20AM (#21728821)
I'd pay a subscription fee given that none of my digital fingerprints is being logged.
Re:Building a Search Engine from Scratch by tschakkaMarc on Saturday December 07, 2019 @04:35AM (#21729335)
Marc from Cliqz here: it’s something I’m thinking about a lot since years, it’s a very clean model. We even did some tests, but unfortunately the majority of people is not like hacker news. What is a reasonable price point for you per month? How would you pay (ie would you feel comfortable giving us your credit card and create an account? (I don’t think crypto is big enough yet to solve this)
Re:Building a Search Engine from Scratch by Animats on Saturday December 07, 2019 @12:43AM (#21728721)
Certainly it can be done. Cuil did it with about 30 people a decade ago. No business model, they ran out of funding, and no one wanted to acquire them, but they did get the crawler and search engine running and publicly available.

It might be worthwhile to do the search part in-house and outsource the question-answering functions. Wolfram Alpha and IBM Watson could be used for answering common questions.

Re:Building a Search Engine from Scratch by stingraycharles on Saturday December 07, 2019 @03:20AM (#21729136)
Doesn’t that imply they weren’t successful?
Re:Building a Search Engine from Scratch by colejohnson66 on Saturday December 07, 2019 @12:12PM (#21731316)
I’m guessing successful in the sense that they had a product come to fruition