News for hackers. Stuff that matters.
Re:Twitter CEO Endorses DuckDuckGo by __ka on Wednesday November 27, 2019 @07:07PM (#21653476)
I have mixed feelings about this. If you recommend DuckDuckGo, Qwant, Ecosia you're essentially recommending Bing. Each of them have to show bing ads, forward the users' IPs as per Bing's terms of service (to fight ad fraud etc). Similarly, if you recommend StartPage, you're recommending Google. Same drill with ads.

I like DuckDuckGo. But unless they build their own search - I think the fundamental problem has not been tackled. There is no good, independent, private search alternative.

[Edit] I am not trying to single out DDG. There are a few replies that either demand for proof (rightfully) or suggest Bing is used only sometimes. I do not have conclusive proof, but have worked on search. Here's a couple of things for the curious:

1. Try to run this test: If you query "what is my ip" in duckduckgo (DDG), or any of the other ones I mentioned, you will notice in the description of one of the top results this IP: 207.46.13.147. It's a BingBot IP [0]. It's a good enough test to spot where results are coming from.

2. Open two browser windows side by side with DDG and Bing. Turn on results for the country you are in in DDG. Look attentively. Try image search.

It is clear that DDG does some re-ranking based on its own, but it's very often the same results.

[0] https://whatismyipaddress.com/ip/207.46.13.147

Re:Twitter CEO Endorses DuckDuckGo by jt2190 on Wednesday November 27, 2019 @09:37PM (#21654198)
You say:

> Each of [Bing’s search partners] have to show bing ads, forward the users' IPs as per Bing's terms of service (to fight ad fraud etc).

But DuckDuckGo says:

> [W]e never share any personal information with any of our partners. The way it works is when we call a partner for information, it is proxied through our servers so it stays completely anonymous. That is, any call to a partner looks to the partner as it is from us and not the user itself, and no user personal information is passed in that process (e.g. their IP address). That way we can build our search result pages using these 100s of partner sources, while still keeping them completely anonymous to you.

https://help.duckduckgo.com/results/sources/

Re:Twitter CEO Endorses DuckDuckGo by AdamSC1 on Wednesday November 27, 2019 @08:02PM (#21653798)
>Each of them have to show bing ads, forward the users' IPs as per Bing's terms of service (to fight ad fraud etc)

The ToS you are referring to is for companies without custom deals in place.

You can read DuckDuckGo's privacy policy here: https://duckduckgo.com/privacy

Re:Twitter CEO Endorses DuckDuckGo by twblalock on Wednesday November 27, 2019 @09:44PM (#21654224)
> I like DuckDuckGo. But unless they build their own search - I think the fundamental problem has not been tackled. There is no good, independent, private search alternative.

There isn't going to be, because it's really hard and expensive to do that. I don't see how any startup could even begin to compete with their own search engine.

The barrier to entry is extremely high: Bing probably would not have survived long enough to be profitable if it had been a startup. Microsoft put a lot of money into building it.

Subscriptions and micropayments would not be a solution for DDG because too few people would be willing to pay. Ad revenue is the only model that has sustained search engines in the long term.

Re:Twitter CEO Endorses DuckDuckGo by judge2020 on Wednesday November 27, 2019 @09:58PM (#21654281)
I see crawls from DuckDuckGoBot so chances are they're trying.
Re:Twitter CEO Endorses DuckDuckGo by alexis_fr on Thursday November 28, 2019 @01:29AM (#21655056)
It would be quite easy to only index Reddit and StackOverflow.
Re:Twitter CEO Endorses DuckDuckGo by arthurcolle on Wednesday November 27, 2019 @09:57PM (#21654280)
Respectfully, I think you are definitely overestimating the cost of building a search engine. Do you have experience in search specifically? Asking in good faith here.

I think you're absolutely right that it's not trivial, but it isn't a completely unsolvable problem only addressable by the major spyware corporations like Facebook, Google, etc. I think you could build a decent search engine in about 6-18 months if all living expenses were paid and you could just lock yourself into your apartment with no distractions, with maybe a team of 1-3 other extremely talented engineers.

Re:Twitter CEO Endorses DuckDuckGo by ummonk on Thursday November 28, 2019 @03:03AM (#21655444)
You will need data on relevant results for a broad swath of queries to fine tune your ranking models against, and will need to spend millions on infrastructure just to crawl and store an index of the entire web.
Re:Twitter CEO Endorses DuckDuckGo by seattle_spring on Wednesday November 27, 2019 @10:25PM (#21654372)
Wouldn't there be a ton of search startups if it were actually this easy?
Re:Twitter CEO Endorses DuckDuckGo by metastart on Thursday November 28, 2019 @01:12AM (#21654996)
It's not so much the difficulty, though yes it is hard, but it's generally extremely difficult to raise capital as a late entrant into an established industry. Unless there's something disruptive. That said, it doesn't take that much capital to build a new search engine with today's computing infrastructure so a few have tried over the past 5-10 years but none were as good as Google (a minimal benchmark). They found a lot of initial interest then died out quickly (sorry I'm forgetting the names, maybe Powerset, Blekko and another). Without doing any "heavy lifting" and also thus not requiring much capital, these so-called "private" search engines have gotten some traction.
Re:Twitter CEO Endorses DuckDuckGo by pier25 on Wednesday November 27, 2019 @11:23PM (#21654565)
I think competing with Google would actually be a more difficult problem
Re:Twitter CEO Endorses DuckDuckGo by arthurcolle on Wednesday November 27, 2019 @10:56PM (#21654465)
The only honest answer to that question depends entirely on one's interpretation of the strong efficient market hypothesis.
Re:Twitter CEO Endorses DuckDuckGo by metastart on Thursday November 28, 2019 @01:06AM (#21654976)
Yes, you're basically right...but the problem isn't building your own search engine, it's in getting search ads privately as well as just being transparent and honest about how your service works -- none of which is true for DuckDuckGo sadly. Their business model is built on sending your personal data to bing when you click on ads on their search results page which link to Bing (moreover they say they never send your IP or personal information to a third-party, but presenting an ad on your search results without disclosing that the ad links to Bing may not "legally" violate that clause but it does in spirit). Ads are also localized so even if they aren't sending your IP address (which according to Bing they are supposed to be doing, but in their terms they claim not to), they're sending your location data. DuckDuckGo further refuses to clarify what data they do send (i.e. how accurate the location data is, what is it even) to retrieve search ads. They aren't transparent about how they work and their business model is fundamentally built on non-private search ads from Bing. It's not "true privacy" nor sincere to say you're not saving any user-related data when you're sending a lot of it to firms like Bing who do save it. The only search engines close to being truly private are epicsearch.in (part of the Epic Privacy Browser) and maybe some small, interesting efforts like private.sh, neither of which have search ads.
Re:Twitter CEO Endorses DuckDuckGo by squiggleblaz on Thursday November 28, 2019 @06:09AM (#21656271)
* Dealing with a proprietary cloud service is never going to offer privacy. If you want to be confident you have privacy, you need to use public, community efforts which you can become part.

* DuckDuckGo could be saying "We have a client from Kyrgyzstan, please give us an ad". Bing could be saying: Here is the heading, the body, and the URL. DDG could be dumping that in the page. Your privacy is not compromised.

* DuckDuckGo could be saying "We have a client from Kyrgyzstan and a client from Brazil and a client from Texas searching for 'plastic sponge bathtub' or 'green sponge bathtub' or 'green sponge beach'" in response to a search for "green sponge beach" from a Brazilian. Bing could be responding with several answers. DuckDuckGo would then send the relevant answer to the client.

* If you click on an ad, you have no privacy, end of story.

Re:Twitter CEO Endorses DuckDuckGo by zepearl on Wednesday November 27, 2019 @08:29PM (#21653922)
Does anybody have any infos about Qwant?

It's the only EU-based search engine with its own indexing engine ( https://en.wikipedia.org/wiki/Qwant )

I don't care if it's EU or not, but is the index real or kind of half-fake? (like "yes, we have an index, but it's 10MBs big...")

I started using Qwant ~2 months ago and it isn't bad: the snippets that are shown in the search results often don't contain the search terms that I used but the links were so far good - but I did mostly lookups for e.g. functions and other reference data - the few times that I challenged it with exact searches its results were similar to the ones of Google, meaning that it didn't return the websites that contained such exact sequences of characters.

Re:Twitter CEO Endorses DuckDuckGo by beefield on Thursday November 28, 2019 @01:34AM (#21655081)
No knowledge, but at least for my searches it has a definite edge against ddg. (I am European.) What I especially like is that in many cases qwant returns nothing if there is no exact match of my query (typically I made a typo.) I kind of like the feeling that the engine tries to search what I tell it to search, not what it thinks I want to search.
Re:Twitter CEO Endorses DuckDuckGo by Moru on Thursday November 28, 2019 @01:49AM (#21655128)
I like to be able to search things I have no idea how to spell. It's pretty easy to make a guess at spelling words and I want to at least have a suggestion below when something is wrong.
Re:Twitter CEO Endorses DuckDuckGo by beefield on Thursday November 28, 2019 @03:34AM (#21655577)
That's as easy as !g
Re:Twitter CEO Endorses DuckDuckGo by zepearl on Thursday November 28, 2019 @01:20PM (#21659247)
> What I especially like is that in many cases qwant returns nothing if there is no exact match of my query

I agree - I admit of having been surprised the first time that it returned "nothing" but in the end I did like the feeling of knowing that it did search what I wanted to search.

Re:Twitter CEO Endorses DuckDuckGo by pedro2 on Wednesday November 27, 2019 @10:47PM (#21654445)
I would like to know too. I read someone on social media alleging they also used Bing behind the scenes, without attribution, but I haven’t researched further.
Re:Twitter CEO Endorses DuckDuckGo by zepearl on Thursday November 28, 2019 @01:24PM (#21659275)
You might be right :(

By searching some random words like "when to buy fruitcakes" (without double quotes) I get almost the same listing in Qwant as in Bing. The results are a bit too similar... .

Re:Twitter CEO Endorses DuckDuckGo by catalogia on Wednesday November 27, 2019 @07:14PM (#21653533)
I'd rather 100 people know 1% about me than 1 person know 100% about me. Even if the only thing duckduckgo did was spread knowledge about me thinner, it would have value. Google already has most of my email (because most people I send emails to use gmail) and my youtube viewing history. Giving them my general search history as well isn't something I'm keen on.

Duckduckgo isn't perfect but they're better than google. If I knew of something better than duckduckgo, I'd use that instead. But I'm not going to let perfect be the enemy of good.

Re:Twitter CEO Endorses DuckDuckGo by liambates on Wednesday November 27, 2019 @07:12PM (#21653519)
Aren't DuckDuckGo searches a mixture of Bing, Yahoo, Yandex and their own crawler?

I don't know a great deal about their infrastructure and would love to learn more about it.

Re:Twitter CEO Endorses DuckDuckGo by mikeleeorg on Wednesday November 27, 2019 @07:38PM (#21653683)
This article is from 2017, but has some more technical detail behind DDG than other articles do:

https://www.wired.co.uk/article/duckduckgo-anonymous-privacy

Re:Twitter CEO Endorses DuckDuckGo by Rebelgecko on Wednesday November 27, 2019 @07:26PM (#21653626)
I don't know about the DDG searches, but IIRC Yahoo is just Bing now
Re:Twitter CEO Endorses DuckDuckGo by fermentation on Wednesday November 27, 2019 @08:00PM (#21653789)
So a mix of Bing and Yandex?
Re:Twitter CEO Endorses DuckDuckGo by diffeomorphism on Thursday November 28, 2019 @03:44AM (#21655621)
Re:Twitter CEO Endorses DuckDuckGo by Yizahi on Thursday November 28, 2019 @07:28AM (#21656766)
There are basically 4 search engines left in the whole world - Baidu (China), Yandex (Russia, recently officially with KGB management), Google and Bing. That is all, full stop. Everyone else is a fancy GUI for on them. Google has more than 90% market share and each of the rest has like 1-3%. Building new engine is prohibitively costly and hard. So the lesser evil is Bing, by simple elimination.
Re:Twitter CEO Endorses DuckDuckGo by JaRail on Sunday December 01, 2019 @03:54PM (#21677724)
You can't include Baidu in the list and say Google has 90% market share. Saying Baidu has 1-3% without specifying "in the USA" is misleading. They've got a billion users in China.
Re:Twitter CEO Endorses DuckDuckGo by st1ck on Thursday November 28, 2019 @02:00PM (#21659485)
AFAIK, DDG uses Yandex engine for searches in Russian.
Re:Twitter CEO Endorses DuckDuckGo by kitotik on Wednesday November 27, 2019 @07:15PM (#21653542)
> But unless they build their > own search

IIUC, they have built their own search, but use Bing(among others) for paid ads.

If this is the case, it seems that it’s the business model that needs innovation and not the search engine itself.

Re:Twitter CEO Endorses DuckDuckGo by __ka on Wednesday November 27, 2019 @07:19PM (#21653578)
Try to run this test. If you query "what is my ip" in duckduckgo, or any of the other ones I mentioned, you will notice in the description of one of the top results this IP: 207.46.13.147. It's a BingBot IP [0]. It's a good enough test to spot where results are coming from.

[0] https://whatismyipaddress.com/ip/207.46.13.147

Re:Twitter CEO Endorses DuckDuckGo by mda on Wednesday November 27, 2019 @08:11PM (#21653848)
Building even a mediocre search engine involves a gigantic index in >40 languages that is constantly refreshed, plus image and video indexes, porn, spam, malware filters, complex ranking, freshness, originality algorithms, constant quality analysis and continuous maintenance and improvements for the infrastructure. They are not transparent about breakdown of queries or details of their deals with Bing and Yandex. They try really hard to create an illusion that it is a real search engine, but seems like it is basically smoke and mirrors.
Re:Twitter CEO Endorses DuckDuckGo by jolmg on Wednesday November 27, 2019 @08:27PM (#21653910)
Wonder how viable it is for a web search engine to support itself on paid subscription. It could also be $X per N search result pages.
Re:Twitter CEO Endorses DuckDuckGo by kelnos on Wednesday November 27, 2019 @10:40PM (#21654430)
> Try to run this test: If you query "what is my ip" in duckduckgo (DDG)

I tried that, and did not see 207.* anywhere on the page. It displayed my IP address above the results list, and then a list of reasonable results for that query, without any IP addresses listed in any of the descriptions.

Re:Twitter CEO Endorses DuckDuckGo by Thorrez on Wednesday November 27, 2019 @10:56PM (#21654467)
I see it in the 3rd result:

https://i.imgur.com/HbhvISk.png

This means DuckDuckGo uses Bing for its results. But it doesn't mean DuckDuckGo is forwarding your IP to Bing.

Re:Twitter CEO Endorses DuckDuckGo by apatters on Wednesday November 27, 2019 @11:56PM (#21654687)
A duopoly will be marginally better at implementing user-friendly features like privacy than a monopoly, because there is at least nominal competition.

However I think we need to start asking deeper questions - why is it that concentration is so prevalent in the software industry?

Software is often said to have "natural" monopoly characteristics, but are those characteristics really a product of "nature" in the way that say the laws of physics are, or are they a product of IP laws and the regulatory choices made by the government?

Re:Twitter CEO Endorses DuckDuckGo by inapis on Thursday November 28, 2019 @03:40AM (#21655595)
I feel it’s because of network effects as well as inability to steal IP easily.

Network effects purely because once you have enough mass of people on board, you can easily attract more people and it is easy enough to distinguish yourself from your competitors that it deters people from switching and/or even being able to make a comparison.

Second you can’t easily steal IP. Anker making a next generation charger will have to face copycats who went to their OEM and were sold the design or the product itself from the backdoor. It’s simply difficult to keep things under wraps when it has a physical + human component. Not so easy in software. To substitute Instagram you’ll probably [0] have to write everything from scratch. Computers when configured correctly will not exactly sell your work from the back door.

[0] I mean source code is still stolen but it’s still marginally harder to use source code than just selling the stolen designs. Plus stealing source code works only once because the victim will then clamp down harder and then continue to innovate it’s way out of the mess.

Re:Twitter CEO Endorses DuckDuckGo by guelo on Wednesday November 27, 2019 @07:58PM (#21653782)
> forward the users' IPs

Seems doubtful since that would be contrary to DDG's main marketing message and their reason for even existing.

Re:Twitter CEO Endorses DuckDuckGo by iagovar on Wednesday November 27, 2019 @07:10PM (#21653501)
AFAIK qwant is using its own engine
Re:Twitter CEO Endorses DuckDuckGo by jlarocco on Wednesday November 27, 2019 @07:20PM (#21653584)
Do you have a reference about DDG using Bing?

It's not mentioned on their site, or their wikipedia entry, or anywhere else I could find, and the results were different when I tried a few searches in them both just now.

Are they "Using Bing" in the same sense they "Use Wikipedia" to put info boxes on the results page sometimes?

Re:Twitter CEO Endorses DuckDuckGo by ac29 on Wednesday November 27, 2019 @07:35PM (#21653675)
"We also of course have more traditional links in the search results, which we also source from a variety of partners, including Verizon Media (formerly Yahoo) and Bing"

https://help.duckduckgo.com/results/sources/

Re:Twitter CEO Endorses DuckDuckGo by godelski on Wednesday November 27, 2019 @07:58PM (#21653781)
That doesn't suggest anything that the OP is saying. Google does similar things. We wouldn't say that searching DDG is like searching Stack Overflow. Or it is like searching Yelp.
Re:Twitter CEO Endorses DuckDuckGo by Thorrez on Wednesday November 27, 2019 @11:00PM (#21654481)
There's a difference between getting results from sites that contain content (Wikipedia, Stack Overflow, Yelp) and sites that are pure search engines (Bing).
Re:Twitter CEO Endorses DuckDuckGo by godelski on Thursday November 28, 2019 @01:45AM (#21655119)
There's also a difference between reskinning a search engine and aggregating other engines' results with your own.
Re:Twitter CEO Endorses DuckDuckGo by Thorrez on Thursday November 28, 2019 @10:26AM (#21657922)
Sure, and AFAIK Google does neither of those, so it would be wrong to say "Google does similar things".
Re:Twitter CEO Endorses DuckDuckGo by godelski on Thursday November 28, 2019 @03:13PM (#21659894)
> AFAIK Google does neither of those

> it would be wrong to say "Google does similar things"

This does not follow.

It also isn't what I was suggesting, but it would surprise me if Google didn't look at what Bing and Yahoo were doing (and DDG).

Re:Twitter CEO Endorses DuckDuckGo by thephyber on Wednesday November 27, 2019 @07:25PM (#21653619)
DDG does use Bing for _some_ of their search results, but it's far from just a reskin.
Re:Twitter CEO Endorses DuckDuckGo by unicornporn on Wednesday November 27, 2019 @08:24PM (#21653888)
DDG uses Bing, Yahoo! and Yandex for non bang searches[1].

As of October 2019, Yahoo! Search is once again “powered by Bing”[2].

To me, results are as crap as Bing's.

[1] https://www.reddit.com/r/duckduckgo/comments/6uaibg/comment/...

[2] https://en.m.wikipedia.org/wiki/Yahoo!_Search

Re:Twitter CEO Endorses DuckDuckGo by godelski on Wednesday November 27, 2019 @09:45PM (#21654227)
"Uses" is a weird word here. Because some people are suggesting that DDG is a reskin of Bing. But if we look at what DDG says[0], they talk about it the same way they talk about Stack Overflow. No one is crazy enough to say that DDG is a reskin of SO. What they are actually saying is that they are aggregating results. Which I don't understand why that would be a bad thing.

[0] https://help.duckduckgo.com/results/sources/

Re:Twitter CEO Endorses DuckDuckGo by Kiro on Thursday November 28, 2019 @05:58AM (#21656219)
Money quoute right here: "We also of course have more traditional links in the search results, which we also source from a variety of partners, including Verizon Media (formerly Yahoo) and Bing."

SO and the other 400 sources are only used for instant boxes and other widgets. The real organic search results are all from Bing.

Re:Twitter CEO Endorses DuckDuckGo by godelski on Thursday November 28, 2019 @03:15PM (#21659902)
Full quote

> In fact, DuckDuckGo gets its results from over four hundred sources. These include hundreds of vertical sources delivering niche Instant Answers, DuckDuckBot (our crawler) and crowd-sourced sites (like Wikipedia, stored in our answer indexes). We also of course have more traditional links in the search results, which we also source from a variety of partners, including Verizon Media (formerly Yahoo) and Bing.

I'm not sure where you're getting

> The real organic search results are all from Bing.

That paragraph __DOES NOT__ say that

Re:Twitter CEO Endorses DuckDuckGo by Kiro on Friday November 29, 2019 @02:01AM (#21662631)
The paragraph clearly says that.

> more traditional links in the search results

It's super obvious to me what that refers to - the actual search results. The rest is just fluff.

Re:Twitter CEO Endorses DuckDuckGo by notus on Wednesday November 27, 2019 @08:50PM (#21654013)
Bing is really good for porn for some reason
Re:Twitter CEO Endorses DuckDuckGo by diffeomorphism on Thursday November 28, 2019 @03:44AM (#21655617)
Uses as some of their main sources:

> In fact, DuckDuckGo gets its results from over four hundred sources. These include hundreds of vertical sources delivering niche Instant Answers, DuckDuckBot (our crawler) and crowd-sourced sites (like Wikipedia, stored in our answer indexes). We also of course have more traditional links in the search results, which we also source from a variety of partners, including Verizon Media (formerly Yahoo) and Bing.

https://help.duckduckgo.com/results/sources/

The results are very similar but not identical to using yahoo or bing.

Re:Twitter CEO Endorses DuckDuckGo by mda on Wednesday November 27, 2019 @07:39PM (#21653692)
_most_ of their search results. They don't give a breakdown.
Re:Twitter CEO Endorses DuckDuckGo by kats on Thursday November 28, 2019 @03:35AM (#21655579)
I may be wrong, but I read before on HN than bing is used for 99% of the search results.
Re:Twitter CEO Endorses DuckDuckGo by CriticalCathed on Wednesday November 27, 2019 @10:11PM (#21654326)
This isn't true about DuckDuckGo. You've come to an erroneous conclusion based on bad interpretation of information.
Re:Twitter CEO Endorses DuckDuckGo by U8dcN7vx on Thursday November 28, 2019 @01:56AM (#21655166)
They survive based on ad revenue. OTOH they provide an ad-free and to my mind a less annoying "lite" interface as well.

<https://duckduckgo.com/lite>

Re:Twitter CEO Endorses DuckDuckGo by randomsearch on Thursday November 28, 2019 @03:16AM (#21655516)
Please edit your post as you’re spreading FUD and are clearly incorrect.
Re:Twitter CEO Endorses DuckDuckGo by Madmallard on Wednesday November 27, 2019 @07:21PM (#21653589)
wtf is that true? ip logging literally defeats the purpose of "privacy"...
Re:Twitter CEO Endorses DuckDuckGo by jv22222 on Wednesday November 27, 2019 @07:25PM (#21653617)
I’m not sure it is true.