News for hackers. Stuff that matters.
Re:Google Search Is Dying by brundolf on Tuesday February 15, 2022 @01:45PM (#30350842)
Free project idea for someone with more time on their hands than me:

Classical search engines determine trust automatically, based on various factors including "link neighborhoods" where trustworthy sites link to other trustworthy sites. These automated strategies are clearly breaking down; the spammers are winning the arms-race.

So maybe we need to go back to human-based trust.

People used to curate lists of websites, which partly solved this problem but didn't necessarily scale. I wonder if that idea could be supercharged.

Consider a browser extension that people install, which:

a) gives users a button to mark a site as trusted/favorited

b) tracks domains visited (and frequency)

Then, separately, you can manually add people you know personally to your "network". You trust them, so anything they trust is also something you might be able to trust. Manual favorites could be weighted higher than frequently-visited sites, and both could be displayed inline next to links on all pages you visit. You could also see which people the trust in a given link comes from, in case some of them consistently have bad judgement about these things and you want to remove them from your list. Then, finally, you could create a personalized search-engine that only indexes the sites determined to be trusted by your personal network.

Of course this would require placing a great amount of trust in the extension and service themselves, so maybe they would have to be open-sourced or self-hostable or something (a profit motive might create a huge amount of temptation to abuse the data). That's a stickier problem.

Edit: There was a little ambiguity left here about transitive trust; “friends of friends” type stuff. I think if this went on for unlimited hops, we’d be back at square one. So maybe it only uses direct contacts, or maybe some small N of hops (where longer ones are weighted lower?). Maybe this would be configurable, not sure.

Also re: privacy, maybe you could come up with a clever way to E2E encrypt the site visit data, even though it’s shared with many parties?

Re:Google Search Is Dying by Seirdy on Tuesday February 15, 2022 @02:00PM (#30351062)
There's no need for an extension; plenty of websites are part of webrings or feature blogrolls. I'm in the process of adding one to mine.

Throw in some microformats2 and/or schema.org structured data and you're good to go.

Certain search engines specialize in this type of manually-curated content; I listed some in the "non-generalist search" section of my collection of indexing search engines: https://seirdy.one/2021/03/10/search-engines-with-own-indexe...

Re:Google Search Is Dying by whoibrar on Tuesday February 15, 2022 @04:17PM (#30352854)
I've just had the deja Vu reading this post! Took me a minute to realise, I did actually come across this post and your blog some time ago, and really enjoyed reading all the posts then and now again! Hoping to read more from you!
Re:Google Search Is Dying by wdencker on Tuesday February 15, 2022 @05:12PM (#30353553)
This is very much in the spirit of what we were trying to do with trove.to [1] — give people an easy way to curate & annotate lists of websites, and layer a social graph and endorsement system on top of those lists.

The problem we encountered is that the vast majority of people are not hyper-organized list makers — the 1% rule of the internet [2]. To create a "human curated search engine" with any utility, you need a massive amount of manually-categorized data — data which most people are simply not interested in generating. This is why no social bookmarking site (e.g. delicious, pinboard, etc.) has ever taken off to hundreds of millions of users.

I still think there's something exciting to be built here, but it will likely need to take a more "automated" approach as you suggested.

[1] https://trove.to/

[2] https://en.wikipedia.org/wiki/1%25_rule_(Internet_culture)

Re:Google Search Is Dying by ImaCake on Tuesday February 15, 2022 @06:03PM (#30354115)
Reddit effectively has this information buried in its database. But they don’t really use it, preferencing issues of the day and recycled tik toks apparently.
Re:Google Search Is Dying by victor9000 on Tuesday February 15, 2022 @07:18PM (#30354802)
Except Reddit is now a top-down company, so I wouldn't expect them to innovate beyond what a typical MBA can regurgitate.
Re:Google Search Is Dying by brundolf on Tuesday February 15, 2022 @05:21PM (#30353665)
Yeah- that’s the only reason I suggested the automatic tracking of visits by domain, despite the obvious privacy complications there
Re:Google Search Is Dying by PoignardAzur on Tuesday February 15, 2022 @03:50PM (#30352507)
The biggest problem, doing that, is that you increase the pressure from bad actors to websites that are trusted by lots of people.

So, for instance, the more weight you give to sites that are quoted by Wikipedia in your search rankings, the more content farms will have incentives to sneak edits that link to their sites.

There are ways to counter that (eg moderation), but in general, defense is more expensive than offense.

Re:Google Search Is Dying by brundolf on Tuesday February 15, 2022 @04:01PM (#30352658)
Right but the idea is that you don’t use sites as sources of trust, you use people. Content farms won’t be paying your friends and family to shill websites to you.
Re:Google Search Is Dying by dangerlibrary on Tuesday February 15, 2022 @04:14PM (#30352807)
I mean, of course they will?

A large fraction of facebook content is MLMs and mobile game ads from one's own contacts.

Re:Google Search Is Dying by nfc on Tuesday February 15, 2022 @04:55PM (#30353332)
Then you just stop trusting them for your search results, it's not like you are unfriending them.
Re:Google Search Is Dying by dangerlibrary on Thursday February 17, 2022 @12:27PM (#30375937)
I think you are applying a solution that would work in a cooperative game. The problem is that this situation, like spam, is an inherently adversarial one - and one where one's adversaries are very motivated and have substantial (substantially more?) resources than you at the outset.
Re:Google Search Is Dying by hutzlibu on Tuesday February 15, 2022 @04:14PM (#30352804)
"Like us on Facebook and get this 10 cent coupon or even the chance of winning something real!"

Works suprisingly well already, sadly. So yes - at some point companies would pay people to vote for them.

But I still would prefer that system (with some differences) over the default. Because the people I would trust, would not fall for the common scams.

Re:Google Search Is Dying by dennis_moore on Tuesday February 15, 2022 @02:39PM (#30351548)
I like this idea of incorporating trust and reputation. As for the curation of websites not scaling, some time ago I thought about the possibility of a search engine where the user supplies a list of trusted websites (for example, university websites, blogs of people they admire), and the search engine ranks pages based on link distance to these websites.
Re:Google Search Is Dying by thrtythreeforty on Tuesday February 15, 2022 @04:44PM (#30353188)
This reminds me of Cory Doctorow's "whuffie" from Down and Out in the Magic Kingdom (man, that title takes me back!).

Whuffie is, roughly, money determined by your social interactions. More importantly, others also have a queryable score that's weighted according to who you esteem highly - this sounds like what you're proposing!

Re:Google Search Is Dying by nfc on Tuesday February 15, 2022 @05:06PM (#30353475)
I was thinking something similar recently, and I also believe it's an idea worth exploring. Something else to add to this conversation. There's an obvious difference between two cases in which a trusted person trusts a url:

1) Single contributor website (blog, personal page...): It seems that we could spread the trust the whole website in the algorithm (at least more than for the next case)

2) Multi contributor website (forum, newspaper): It seems the trust should be given at an URL level

Something worth delving into if we are designing this trust based search engine in real-time here at HN ;)

Re:Google Search Is Dying by allochthon on Tuesday February 15, 2022 @04:53PM (#30353302)
This was kind of the idea behind a side project I started a few years ago:

https://blog.digraph.app/2020-06-13-democratization-of-searc...

In that post, I don't address the reputation management aspect as much, but it's central to making the whole thing work, and I think crowd-sourcing and a well-conceived reputation management system that can influence results are good next areas for exploration.

Re:Google Search Is Dying by Corrado on Tuesday February 15, 2022 @02:44PM (#30351632)
I think the Keybase project would have been great for providing the "authentication" part of this solution. To bad it died on the vine of the Zoom purchase.
Re:Google Search Is Dying by eitland on Tuesday February 15, 2022 @02:04PM (#30351101)
Someone will sooner or later mention the sybil attack.

(At some points I feel it is thrown out as haphazardly as "correlation does not imply causation".)

But I think you might be onto something. It won't necessarily be easy but I think it deserves more than a quick dismissal.

Re:Google Search Is Dying by throw10920 on Tuesday February 15, 2022 @02:41PM (#30351585)
GP comment already explicitly addressed this:

> Then, separately, you can manually add people you know personally to your "network". You trust them, so anything they trust is also something you might be able to trust.

It doesn't matter how many sockpuppet accounts a marketing company creates if you don't click the "trust" button on any of them, and if the system is designed so that your local trust network only consists of what you trust directly and (weighted) transitively.

Re:Google Search Is Dying by foxfluff on Tuesday February 15, 2022 @04:15PM (#30352828)
I've been wanting[0] more or less the same for a long time now. Ideally the dataset should be public (and distributed) so that you could find bad actors and eliminate them from your network (i.e. when you see a bad result, you can figure out where it came from into your feed), as well as run your own tweaked algorithms. WoT alone is great, but it gets better still when you can tweak the algorithm.

At that point, it doesn't need to be people you know personally; it just needs to be identities with a proven track record. If you can manage multiple identities, you could make ones e.g. for the explicit purpose of presenting a blacklist of bad sites you've found, or for providing a "front page" to a certain subject. Anyone is free to trust your identity or not, and choose exactly how to rank its actions.

This whole thing could be expanded to tagging and other metadata as well as comments, and so on.

[0] https://news.ycombinator.com/item?id=29691303

Re:Google Search Is Dying by oehpr on Wednesday February 16, 2022 @01:16AM (#30356895)
I'm so glad to see others talking about this.

A system of distributed trust like this is the makings of decentralized moderation. Decentralized curation. I've also had this idea on my mind for a while: https://news.ycombinator.com/item?id=21975294

Re:Google Search Is Dying by batisteo on Tuesday February 15, 2022 @03:57PM (#30352603)
Like the Web-of-Trust or similar?
Re:Google Search Is Dying by sdeframond on Tuesday February 15, 2022 @01:57PM (#30351015)
Interesting.

Another "sticky" problem is how to make a living out of this I guess...

Re:Google Search Is Dying by rsyring on Tuesday February 15, 2022 @02:02PM (#30351078)
Have the extension serve ads.
Re:Google Search Is Dying by JamesBarney on Tuesday February 15, 2022 @05:02PM (#30353422)
You could also incorporate "reputation" of the author. Basically have a real person, the author stake their real identity on the quality of the blog post they wrote.