Classical search engines determine trust automatically, based on various factors including "link neighborhoods" where trustworthy sites link to other trustworthy sites. These automated strategies are clearly breaking down; the spammers are winning the arms-race.

So maybe we need to go back to human-based trust.

People used to curate lists of websites, which partly solved this problem but didn't necessarily scale. I wonder if that idea could be supercharged.

Consider a browser extension that people install, which:

a) gives users a button to mark a site as trusted/favorited

b) tracks domains visited (and frequency)

Then, separately, you can manually add people you know personally to your "network". You trust them, so anything they trust is also something you might be able to trust. Manual favorites could be weighted higher than frequently-visited sites, and both could be displayed inline next to links on all pages you visit. You could also see which people the trust in a given link comes from, in case some of them consistently have bad judgement about these things and you want to remove them from your list. Then, finally, you could create a personalized search-engine that only indexes the sites determined to be trusted by your personal network.

Of course this would require placing a great amount of trust in the extension and service themselves, so maybe they would have to be open-sourced or self-hostable or something (a profit motive might create a huge amount of temptation to abuse the data). That's a stickier problem.

Edit: There was a little ambiguity left here about transitive trust; “friends of friends” type stuff. I think if this went on for unlimited hops, we’d be back at square one. So maybe it only uses direct contacts, or maybe some small N of hops (where longer ones are weighted lower?). Maybe this would be configurable, not sure.

Also re: privacy, maybe you could come up with a clever way to E2E encrypt the site visit data, even though it’s shared with many parties?

Throw in some microformats2 and/or schema.org structured data and you're good to go.

Certain search engines specialize in this type of manually-curated content; I listed some in the "non-generalist search" section of my collection of indexing search engines: https://seirdy.one/2021/03/10/search-engines-with-own-indexe...

The problem we encountered is that the vast majority of people are not hyper-organized list makers — the 1% rule of the internet [2]. To create a "human curated search engine" with any utility, you need a massive amount of manually-categorized data — data which most people are simply not interested in generating. This is why no social bookmarking site (e.g. delicious, pinboard, etc.) has ever taken off to hundreds of millions of users.

I still think there's something exciting to be built here, but it will likely need to take a more "automated" approach as you suggested.

[1] https://trove.to/

[2] https://en.wikipedia.org/wiki/1%25_rule_(Internet_culture)

So, for instance, the more weight you give to sites that are quoted by Wikipedia in your search rankings, the more content farms will have incentives to sneak edits that link to their sites.

There are ways to counter that (eg moderation), but in general, defense is more expensive than offense.

A large fraction of facebook content is MLMs and mobile game ads from one's own contacts.

Works suprisingly well already, sadly. So yes - at some point companies would pay people to vote for them.

But I still would prefer that system (with some differences) over the default. Because the people I would trust, would not fall for the common scams.

Whuffie is, roughly, money determined by your social interactions. More importantly, others also have a queryable score that's weighted according to who you esteem highly - this sounds like what you're proposing!

1) Single contributor website (blog, personal page...): It seems that we could spread the trust the whole website in the algorithm (at least more than for the next case)

2) Multi contributor website (forum, newspaper): It seems the trust should be given at an URL level

Something worth delving into if we are designing this trust based search engine in real-time here at HN ;)

https://blog.digraph.app/2020-06-13-democratization-of-searc...

In that post, I don't address the reputation management aspect as much, but it's central to making the whole thing work, and I think crowd-sourcing and a well-conceived reputation management system that can influence results are good next areas for exploration.

(At some points I feel it is thrown out as haphazardly as "correlation does not imply causation".)

But I think you might be onto something. It won't necessarily be easy but I think it deserves more than a quick dismissal.

> Then, separately, you can manually add people you know personally to your "network". You trust them, so anything they trust is also something you might be able to trust.

It doesn't matter how many sockpuppet accounts a marketing company creates if you don't click the "trust" button on any of them, and if the system is designed so that your local trust network only consists of what you trust directly and (weighted) transitively.

At that point, it doesn't need to be people you know personally; it just needs to be identities with a proven track record. If you can manage multiple identities, you could make ones e.g. for the explicit purpose of presenting a blacklist of bad sites you've found, or for providing a "front page" to a certain subject. Anyone is free to trust your identity or not, and choose exactly how to rank its actions.

This whole thing could be expanded to tagging and other metadata as well as comments, and so on.

[0] https://news.ycombinator.com/item?id=29691303

A system of distributed trust like this is the makings of decentralized moderation. Decentralized curation. I've also had this idea on my mind for a while: https://news.ycombinator.com/item?id=21975294

Another "sticky" problem is how to make a living out of this I guess...

Different website. Different title. Exact same content. 4 or 5 in the first page of search results.

I'm assuming they're all ran by the same person, throwing as much ** at the wall knowing some will stick.

Many of my searchers now include "reddit" or "forum" at the end to filter out all the spam/crap.

But the web, in the sense of quality:crap ratio, has gotten substantially worse.

This flood seems like the ultimate manifestation of turnkey hosting solutions.

Imho, we could do worse than reviving an idea from email's early days vs spam: negligible per-use charging. The idea was to tax emails at $0.0001 (or somesuch). Insignificant for actual users, but financially decimates high-volume, low-value spammers.

Nobody is producing real content that isn't behind a paywall.

There's nothing to find.

Google search is constantly improving but the SEO spammers are improving faster.

If there were no Google though, we'd likely have the same thing.

So I guess the only reality that avoids incentivizing them is one where (1) there is a massive traffic generator & (2) that massive traffic generator severely disincentizes content farms.

I'm surprised they haven't done some kind of manual pruning of junk like that, or maybe they have and it's not working... but on the surface it totally seems like they could implement something that says "GitHub has content X, and these other 10 sites are 99% the same, but we've flagged GitHub as an authoritative source so they'll always outrank the clones".

Maybe it's a fear of appearing unfair. Or maybe they secretly want to hurt Microsoft by turning a blind eye. Or maybe this is actually a much harder problem. If I had to guess it's probably #3. But as a user of search it's frustrating to find the clones ranked above the real stuff.

Meanwhile I have in my inbox in the last 24h at least a half-cozen emails looking to do SEO work for my company website.

Web = untrustworthy? YUP

I'd happily pay for a serious version of 1999 Google, but updated to filter out anything advert based, and search for exactly what I want.

Search is such a fundamental function, and we've done the experiment and the advert model fails - it needs to be just another utility.

If only google was smart enough to figure this out

https://twitter.com/jdgoesmarching/status/149367886211437772...

Any Googlers want to chime in?

I've clicked on a few out of curiosity, immediately recognizing they were garbage from the description text, and it's just endless SEO links and completely random text.

You'd think one of the richest companies on earth could make a freshman intro to CS-level spam filter. If they can't, then they truly do hire the most incompetent people on earth. If they won't, then they hire the evilest people on earth.

The difference in speed between running a simple script and reading a file is not large enough to noticeably affect SEO.

Most of these sites are probably just using PHP, not any static site generator.

I was recently trying to troubleshoot a very basic error message for Linux and was getting results and webpages that would list the error message in the title in some way, but then give instructions on "First, open up device manager", "Click Win+R to open windows command prompt", etc. Lots of untrustworthy ads. Different URLs, almost line-for-line identical webpages.

This was something like the top four search results (that weren't sponsored ads).

1) if this is from the project's own site, it's good.

2) if it looks like an archive of the project's mailing list, it's good.

3) if it looks like an internet forum, it might be good, or it might be just another poor soul asking the same question.

4) if it's on StackExchange, it's like on the forum, except your chances are slightly better. Karma must flow.

5) if it's on Reddit, it's like on the forum, except your chances of getting an answer are worse.

6) if it's a blog of some geek, sometimes it can be better than 1) and 2), or you might just get a straight answer.

7) in any other case it's most likely a SEO farm. Run.

If I have a linux problem these days, google usually gives me the relevant piece of source code on Github from which the error message originates. Like, you're a big boy now, go figure it out yourself.

It seems we're back to the early 2000s, when search engines not so much specialized in topics, but leaned heavily towards one type of content or the other. Holy hell, maybe one day Reddit's own search will be good enough so google can be ditched for good!

I don't want to see the same result repeated 5 times across different stack overflow mirrors.

Nothing wrong with that but that also indicates the usual clickfarm spammers from developing countries had unfiltered access to Clubhouse from day 1.

This might sound elitist but it's probably a good idea that C2C apps get their first batch of users and community leaders from high income countries before branching out to the rest of the world.

>This might sound elitist but it's probably a good idea that C2C apps get their first batch of users and community leaders from high income countries before branching out to the rest of the world.

Why?

Unless they're targetting local communities with some sort of culturally relevant format, its good for their bottom line to target the most profitable customers of advertising anyways.

Like, who really needs a voip protocol pretending to be a program?

Because doing a search in a place that’s not moderated by humans would generate too much noise.

I think this kinda takes us back to the old times with Yahoo (and humans sorting the information) etc…

A giant step backwards, if you ask me.

I had to search crates.io . Let me tell you. It's not the pinnacle of search.

What I searched was `fast bitset implementation`. My results consisted of drill bits a stack overflow questions and a Baeldung article on hashSet vs long[]

This seems like a natural result of optimizing each search for revenue. Think of a search to solve an error message on your computer. There's a very small number of vulnerable people who are going to spend money as a result of that search, so optimizing for ads would mean tailoring the results specifically for those people, pushing them to sleazy sites where they might spend money on some kind of antivirus scam. The results are worthless to you, but who cares? You're worthless to Google when you're doing that kind of search. Try searching for something that people in your demographic spend money on, and the results will likely look better to you.

I do this to search for items on craigslist across the country.

This will force a reddit search.

Combined with AI imitating speech and deepfakes, and technology of inplanting false memories, we will have the matrix, just not the wau we expected ^ ^