News for hackers. Stuff that matters.
Re:“YouTube-dl” and “Pirate Bay” back on DDG by nomilk on Sunday April 17, 2022 @02:06PM (#31062847)
Why does DDG de-list things in the first place; isn't it in the interest of search engines to be as useful as possible to users, and thus maximise the results provided?

Also curious to know the extent to which Google de-lists things?

Long term, perhaps a decentralised search engine could get around de-listing and provide a more reliable and rigorous search experience.

Re:“YouTube-dl” and “Pirate Bay” back on DDG by RedBeetDeadpool on Sunday April 17, 2022 @05:13PM (#31064576)
DDG CEO is clearly biased. Its no longer a neutral platform:

https://twitter.com/yegg/status/1501716484761997318?s=20&t=9...

Before someone chimes in, yes I understand the humanitarian perspective. That's not my point. My point is that DDG is not neutral, and is politically biased.

Re:“YouTube-dl” and “Pirate Bay” back on DDG by RedBeetDeadpool on Sunday April 17, 2022 @05:36PM (#31064708)
BTW, if you wanted a biased platform, just use Google, its significantly better in every way.

Ultimately what I'm getting at is: There's no market for DDG. Use Google for biased searches, and use other search engines that are not biased(which excludes DDG) for unbiased searches.

What are the use cases for DDG?

Re:“YouTube-dl” and “Pirate Bay” back on DDG by CameronNemo on Sunday April 17, 2022 @08:10PM (#31065842)
Few people switched to DDG because of censorship, the main DDG selling point is privacy. All of their marketing stresses that aspect.
Re:“YouTube-dl” and “Pirate Bay” back on DDG by cinquemb on Monday April 18, 2022 @02:27AM (#31067995)
"All"[0] in the revision of historical examples sense. Unless one really thinks that filter bubbles have no overlap with censorship.

Maybe i was only the few that cared more about the filter bubble angle and less the "we're selling bottled privacy™" angle [1] and am not interested now that `yegg` has clearly reneged upon the former

[0] https://techcrunch.com/2011/06/20/duckduckgo-to-google-bing-...

[1] https://pictobar.tumblr.com/post/63785124046/the-banality-of...

Re:“YouTube-dl” and “Pirate Bay” back on DDG by leach on Sunday April 17, 2022 @09:47PM (#31066437)
Yeah DDG is just a worse google. If I want to use google I might as well use google.

When I want unbiased searches I've been using Kagi but more are popping up. they approach search differently so it's useful when google feels to "sanitized" for certain searches

Re:“YouTube-dl” and “Pirate Bay” back on DDG by ikt on Sunday April 17, 2022 @08:59PM (#31066163)
> BTW, if you wanted a biased platform, just use Google, its significantly better in every way.

It's not to me, when was the last time you used it? I haven't used Google in around 4 years now and I'm getting on just fine, majority of searches answered on first page.

If anything I now toggle between Duckduck and Ecosia as Ecosia still isn't 100% up to scratch (frequent 500 errors, slow, results are bad) but I like the idea of my searches planting trees.

Re:“YouTube-dl” and “Pirate Bay” back on DDG by remram on Sunday April 17, 2022 @09:09PM (#31066225)
Recently Google started ignoring double-quotes entirely which makes it pretty much useless for most of my daily search needs.
Re:“YouTube-dl” and “Pirate Bay” back on DDG by pmoriarty on Sunday April 17, 2022 @05:55PM (#31064849)
DDG respects privacy, or at least claims to. Google has nothing but contempt for it, and thrives on spying on you.
Re:“YouTube-dl” and “Pirate Bay” back on DDG by Germstorm on Monday April 18, 2022 @01:47AM (#31067814)
I use them as default search on my Android devices, to avoid the cookie banner of Google in private browsing mode.
Re:“YouTube-dl” and “Pirate Bay” back on DDG by bedros on Monday April 18, 2022 @01:00AM (#31067609)
I removed DDG from my browsers after his tweet. they are no different from Google
Re:“YouTube-dl” and “Pirate Bay” back on DDG by jbay808 on Sunday April 17, 2022 @02:16PM (#31062925)
I wonder how a decentralized search engine would fight its inevitable SEO war, should it become successful.

I'd expect that all sites wanting to draw traffic would attempt to grab the reins of the search engine to point toward themselves, and the result would be search results ordered by rein-grabbing power.

Not that centralized search engines are immune to this; they're almost as vulnerable (seeing as sponsored search results exist) but the maintainer at least has to balance that with the utility of the search engine overall, to prevent the search engine from falling out of favour.

With a decentralized engine, parties that have deeply invested in manipulating the results will still want the engine to be popular too, but I'm not sure how you resolve the prisoners dilemma there as a whole.

Re:“YouTube-dl” and “Pirate Bay” back on DDG by forgotmypw17 on Sunday April 17, 2022 @02:52PM (#31063218)
> I wonder how a decentralized search engine would fight its inevitable SEO war, should it become successful.

> I'd expect that all sites wanting to draw traffic would attempt to grab the reins of the search engine to point toward themselves, and the result would be search results ordered by rein-grabbing power.

I would venture to say that a combination of allow-lists and block-lists from trusted parties, ranked using some kind of distributed web-of-trust system would work reasonably well.

Re:“YouTube-dl” and “Pirate Bay” back on DDG by KMag on Sunday April 17, 2022 @03:57PM (#31063888)
Ages ago, I was a professional P2P developer, and I vaguely remember some of the research papers on P2P censorship-resistant reputation systems. Generally, you have public signing keys that sign ratings (say, -1.0 to 1.0) for both content and other raters. These signed ratings collections are then pushed into a distributed hash table.

The basic idea is when you rate something, your client also looks up in the DHT other people who have rated the same content with similar ratings. Your client then pulls the latest ratings collections from those people, and computes the cosign distance between your ratings and their ratings (over the intersection of content that both of you have rated). Periodically, your client signs and publishes an updated ratings document, where the rating for other raters is the cosign distance. The cosign distance, the size of the ratings intersection set, and maybe some other factors go into deciding which raters get published out in your ratings update.

When you query for the rating for a given piece of content, your client grabs the list of ratings for that content from the DHT. It then pulls the latest ratings published by those raters, computes cosign distance, and then does something similar to Djikstra's shortest-path algorithm to recursively search the DHT using these cosign distances as weights. In general, the DHT wouldn't have many signatures stored under the content's hash, but by recursively following the graph of other raters, your client hopefully finds other raters that rate things similarly to you and have rated this content. The path weight to a given rater is the product of cosign distances, and so by using a priority queue for querying, you get something close to a breadth-first search of the ratings graph. Once your client has accumulated enough weight of ratings for the given content, it stops and shows you the weighted average of the ratings (and maybe the weighted std. dev. is displayed as a confidence score to power users who have enabled it).

Presumably, the UI for the ratings system maps 0 to 5 stars to 0.0 to 1.0 (probably not linearly, more likely the client locally keeps a histogram of the user's ratings and then maps the star rating back to a percentile rating), and the "spam" button rates the content as -1.0.

The tricks come down to the metrics used for how the DHT decides priorities for cache eviction of the per-content ratings and also the per-rater ratings. You don't want spammers or other censors to be able to easily force cache eviction. Getting cache eviction metrics right is the key to having the system scale well while also preventing spammers/censors from evicting the most useful sets of ratings.

Re:“YouTube-dl” and “Pirate Bay” back on DDG by sweetbitter on Monday April 18, 2022 @03:23AM (#31068231)
What books and/or papers could you recommend for learning all about this, distributed hash tables, cosign distance, reputation systems, the whole deal?
Re:“YouTube-dl” and “Pirate Bay” back on DDG by notatoad on Sunday April 17, 2022 @02:15PM (#31062913)
DDG uses the Bing index. If Bing de-lists something, then it disappears from DDG.

Microsoft (and any other big company) has many competing interests other than just being helpful to users.

Re:“YouTube-dl” and “Pirate Bay” back on DDG by nomilk on Sunday April 17, 2022 @02:19PM (#31062945)
> [big companies have] many competing interests

Are the main ones i) reducing competition and ii) managing their reputation?

If so, the case for a decentralised search engine got stronger.

Re:“YouTube-dl” and “Pirate Bay” back on DDG by notriddle on Sunday April 17, 2022 @05:16PM (#31064594)
> Why does DDG de-list things in the first place

I don’t think anyone wants to use a search engine that never delists anything. Ransomeware, Markov chain junk, plagiarism. A search engine that never delists anything is useless.

The problem is when delisting is used against the end-user’s interests.

Re:“YouTube-dl” and “Pirate Bay” back on DDG by anjbe on Sunday April 17, 2022 @11:55PM (#31067238)
Early on, a major selling point of DuckDuckGo was that it filtered out SEO parked domain pages.
Re:“YouTube-dl” and “Pirate Bay” back on DDG by matheusmoreira on Sunday April 17, 2022 @02:29PM (#31063029)
Copyright industry pressure.
Re:“YouTube-dl” and “Pirate Bay” back on DDG by WhyNotHugo on Tuesday April 19, 2022 @05:38AM (#31081184)
> Why does DDG de-list things in the first place [...]

They use Bing as an index, and it was Bing who de-listed it.

Re:“YouTube-dl” and “Pirate Bay” back on DDG by tdiff on Sunday April 17, 2022 @02:19PM (#31062947)
Banning Russian news outlets from search results was generally supported here on HN.
Re:“YouTube-dl” and “Pirate Bay” back on DDG by ghostly_s on Sunday April 17, 2022 @05:09PM (#31064544)
That's certainly not how I remember that thread going.
Re:“YouTube-dl” and “Pirate Bay” back on DDG by gscott on Sunday April 17, 2022 @02:22PM (#31062966)
There is no war just a special operation.
Re:“YouTube-dl” and “Pirate Bay” back on DDG by tonguez on Sunday April 17, 2022 @04:10PM (#31064007)
we must condemn russias unprovoked genocide of neo nazis
Re:“YouTube-dl” and “Pirate Bay” back on DDG by suction on Sunday April 17, 2022 @06:48PM (#31065281)
Wait, did Russians start committing suicide?