As I understand it, you'd want to continue to search the whole "unbiased" web, then apply different filters / weights on every search. I really do like the idea, but I imagine we'd be talking about an increase in compute requirements of several orders of magnitude for each search as a result.

Maybe something like this could be made a paid feature, with a certain set of reasonable filters / weights made the default.

Maybe this would require too much data to be sent to the client, compared to the usual case where it only needs a page of results at a time. If so, would a compromise be viable, whereby the client receives the top X results and filters those?

The problem is that we have no say in ranking and filtering. I think it should be customisable both on a personal and community level. We need a way to filter out the crap and surface the good parts on all these sites. I am sure Google wouldn't like to lose control of ranking and filtering, but we can't trust a single company with such an essential function of our society, and we can't force a single editorial view on everyone.

As we have many newspapers, each with its own editorial views, we need multiple search engine curators as well.

I'd also love to be able to specify I want results from the last year without having to set it everytime.

I got signed up for goodreads (book review site), and I get tons of spam. It's not quite the same as your idea, but it is a currated list. I don't know how you stop spammers from adding bogus links in the python interest list (to use an example).

This is a hard problem..

EDIT: Clarified goodreads reference!

pro-privacy does not sit well with terms such as search history and user profile

But a system such as I'm describing is probably the only one that can be entirely consistent with the two disparate requirements of fully anonymizing users, and being useful to both programmers and ophiologists studying different things called "python".

It would basically be like sending a search query in this form:

"Python importerror help --prefs={mit_cs_club, studioghiblifans_new, britains_best_baking_prefs, AlpineMountaineersIntl}"

If you like baking, anime, and mountaineering, it's probably convenient to leave all those active for your searches, even your purely programming-focused searches. But you could toggle some of them off if articles about "helping to protect imported mountain pythons" are interfering with your search results, or if you want to be more anonymous. If you're especially paranoid you could even throw in a bunch of random preferences that don't affect your query but do throw off attempts to profile you. You could pretty easily write a script that salts every search with a few extra random preference lists, for privacy or just for fun, and make that an additional feature. The tool doesn't need to maintain any history of your past activity to cater to your search, so I think it would be a good thing for privacy overall.

the more anonymous among us tend to opt for common IP addresses and common user agents to become the tree among the forest. adding a profile to that would, well, only add to a digital fingerprinting profile

> those preferences don't need to be attached to you, just your query

that's not how it works. preferences are by their nature personal. every transaction would have your interests and hobbies embedded, on top of metadata

> mit_cs_club, studioghiblifans_new, britains_best_baking_prefs, AlpineMountaineersIntl

you have voluntarily made yourself the birch among the ebony

> If you're especially paranoid you could even throw in a bunch of random preferences that don't affect your query but do throw off attempts to profile you

how would they not affect the query? they complement the query. or rather, unnecessarily accompany the query. your results depend on your input. it doesn't matter what colour glove you wear to pull the trigger if you bury the gun with the body

> write a script that salts every search with a few extra random preference lists, for privacy or just for fun

just the latter. fuzzing would be pointless since the engine will have already identified you by now

it sounds like an annoying browser extension at best. to label it a pro-privacy tool would be ludicrous

So I think that stepping back and re-thinking what a search engine fundamentally is, is a great starting point for disruption.

Additionally, something the OP didn't mention is that ML technologies have progressed dramatically since 1998, and that much of that progress has been done in the open. I can't imagine that not being a force-multiplier for any upstart in this domain.

As such, they effectively have a list of "tier 1" domains.

> Second, we measure engagement of specific events on the page (e.g. when a misspelling message is displayed, and when it is clicked). This allows us to run experiments where we can test different misspelling messages and use CTR (click through rate) to determine the message's efficacy. If you are looking at network requests, these are the ones going to the one-pixel image at improving.duckduckgo.com. These requests are anonymous and the information is used only by us to improve our products.

The Firefox network logger does show requests to this domain when I click on a link in the search results, before the page navigates away. This suggests to me they might by logging this information. To be clear, this is speculation on my part, because I haven't examined the URL parameters in detail.

In any case, I'm not sure how much this manages to improve the results, since usually I can get help with my Python query (for example) using whatever crappy blog post is first in the results, but results from the official docs or StackExchange are still probably better and should be prioritized.

Any system that ranks things purely based on votes or view counts can have a feedback loop that can amplify "bad" results that happen to get near the top for whatever reason. For web search, this would encourage results that look right from the results page, even if they're not actually a good result of what the user is looking for.

An example of this would be when you're trying to find an answer to a specific question like "How do I do X when Y?". The best result I'd hope for is a page that answers the question (or a close enough question to be applicable), while the promising-looking-but-actually-bad result is a page where someone asks the exact same question but there are no answers.

I think this is a place where Google has pretty obvious algorithm problems. For example, I’m building a personal website for the first time in many years, and obviously that means I’m doing a fair bit of looking up new or forgotten webdev stuffs. It’s widely known that W3Schools is low quality/high clickbait/has a long history of gaming the SEO system. They’ve been penalized by Google’s algorithm rule changes but continue to get the top result (or even the top 3-5 results!), even with Google having a profile of my browsing habits, and knowing that I intentionally spend longer on these searches to pick a result from MDN or whatever. It seems pretty likely that W3Schools is just riding click rate to stay at the top. And it’s pathological.

for some languages, W3schools is as good a reference or better than the official documentation.

And they're definitely better than most seospam.

Edit: I semi-intentionally forgot to mention Stack Overflow because it’s so unpredictable in terms of quality.

There is a DuckDuckGoBot and I think it was an interview or podcast Gabriel did a while back that he mentioned they use it for filling out gaps in the Bing API data to provide the instant answers, favicons. Their preference for the instant answers were authoritative references such as docs.python.org. This would have been a while back though.

The problem with this strategy is always going to be that different users will regard different sources as most desirable.

For example, it's enormously frustrating that searching for almost anything Python-related on DDG seems to return lots of random blog posts but hardly ever shows the official Python docs near the top. I don't personally think the official Python docs are ideally presented, but they're almost certainly more useful to me at that time than some random blog that happens to mention an API call I'm looking up.

On the other hand, I would gladly have an option in a search engine to hide the entire Stack Exchange network by default. The signal/noise ratio has been so bad for a long time that I would prefer to remove them from my search experience entirely rather than prioritise them. YMMV, of course. (Which is my point.)

With that logic, Apple’s OCSP server is also 100% anonymous (which I legitimately can believe it is).