I think what I'd say in defense is that we've misunderstood what search engines are useful for. They're really bad at helping us discover new things. Your blog might be awesome, but it's not going to be easy for a search engine to tell that it's awesome. It's going to have to compete with other blogs that also want views, some of whom are going to be better than yours at SEO, and so on.

What a search engine might be able to tell is that it's useful. Because what search engines are at least potentially good at is answering questions. You do that by having a list of known good sites to answer specific types of questions, and looking at the sites they link to. It's when you try to do both (index everything on the web and provide accurate answers to specific questions) that you end up failing to do either. For example this is the #2 result for "python f strings" on DDG[1]. It's total garbage, and, quoting the blog, "we can do better". (This result is also on page 1 for the same query on Google.)

What I believe ddevault is suggesting is that we make a search engine that does the only thing search engines are really good at, answering questions. You throw away the idea of indexing everything on the web, and therefore the possibility of "discovery". What that means is that in 2020 you need some other mechanism for discovering new sites, bloggers, and so on. Fortunately we do have some alternatives in that space.

To be clear, I don't know if I 100% buy this argument, but I think it's the general idea behind what's being suggested in this blog post.

[1] https://careerkarma.com/blog/python-f-string/

It definitely seems that search engines can’t find new websites for people. Now they are just aggregating Q&A.

There are other options that may be better, but in general, very few people are looking for them.

Here's your "python f strings" query on Runnaroo:

https://www.runnaroo.com/search?term=python+f+strings

I'm the creator of Runnaroo.

So you need to speed it up, but it does look good for searching documentation at least.

I mean, that's basically the core of original Google pagerank, right? A "good" site linking to another site is what makes that other site some amount of "good" too, links from better sites carry more 'juice'. "good" is of course not just binary, but a quantitative weight.

I don't know to what extent that's still at the core of their relevancy rankings. I don't know how all those annoying spammy recipe blogs or content farms get to the top of the results either. I don't think it's because Google's engineers believe they are "good" results.

Relevancy ranking on web search is clearly a hard problem, mainly because so many authors are trying to game it, it's a feedback loop.

If Google can only do as google does despite pouring a whole lot of money into it, I don't see a reason to bet that better will be what's basically an over-simplified description of how Google started out doing it (and then evolved it because it wasn't good enough).

I hope the project of improving on DuckDuckGo is successful though, and some of the other proposals in the article sound promising to me.

In the future I would like to see an open source search engine 'paid for' using some combination of homomorphic encryption, blockchain and Tor-style technologies to trade bandwidth and processing power with its userbase in exchange for search results and other services, but I don't have the expertise to assess how feasible that might be.

There could be a community of Software developers running one instance of the OS search engine that focuses on programmers needs: documentation, vcs hosters, dev blogs, tech news websites and on topic blogs. Great if you need to search for how to solve a software issue, terrible if you need to figure out how long to cook spaghetti.

The lists of crawled pages i guess would be visible/searchable too, at least on instances that you could trust.

If you don't like the list, and the maintainers for whatever reason don't want to change it to your liking, feel free to use another instance or set one up yourself.

I find it helps when looking for obscure info on random topics.

Still, it's good to remember that it was once uncertain whether people should access the web using something a table of content (portal/directory) or something like an index (search engine). It seems the search engines won.

See: https://en.wikipedia.org/wiki/DMOZ

You could have user compiled lists of sites to show in search results.

Let the users pick the lists they want to see, and communities can create and distribute lists within themselves.

Could easily be implemented for any current search engine.

To a large extent, this is what you already do when you view a page of search results. Filter them based on your understand of what sites / results hold value.

On a public scale, you could make an argument for tighter integration/better privacy with the lists. For example:

    Browser -----Request-to-SE-----> Search Engine
      ^                                   |
      |                      Unfiltered Results (In YAML/JSON)
      |                                   |
      |                                   V
      |--Desired Results------ Local Filtering/Rendering

Edit: Possibly this could be easier to use to set up distributed search as well, as each node could index a given list, and then distribute that list similarly to DNS. Don't really know how well that would work though, just an idea.

I'm thinking about playing around with this in my spare time, but that part seems the hardest to do.

It’d take a lot of cycles while small but if you get a network growing you could even have sub-networks with their own whitelist additions (and every user has a blacklist.)

It would be bad if those in the positions profit by "authorizing" who is good though.

You can have really biased technically terrible filters that for example put a site on level 4 because it is to new, to small and any number of other dumb SEO nonsense arguments. (The topic was not in the url! There was poor choice of text color!)

I think wikipedia has a lot of research to offer on what to do but also what not to do. Try getting to tier 2 edits on a popular article? It would take days to sort out the edits and construct by hand a tier 2 article.

I expected that to evolve to get more specificity but things went completely the otherway and we can't even specify a term is on a page reliably with Google now.

Similarly, I was all in on xhtml and semantics (like microformats) where you'd be able to search for "address: high street AND item:beer with price:<2" to find a cheap drink.

I imagine for a FOSS solution we would have to make configurable every separable ranking algo and the option to toggle them in groups as well as build cli like queries around them (with a gui)

I'm starting to see a picture now. In stead of wondering how to build a search engine we should just build things that are compatible. A bit like The output of your database is the input of my filter.

Take site search, it is easy to write specs for with tons of optional features and can easily outperform any crawler. Meta site search can produce similar output. Distributed diy cralwers can provide similar data.

Arguably top websites should not be indexed at all. They should provide their own search api.

The end user puts in a query and gets a bunch of results. It goes into a table with a colum for each unique property. The properties show up in the side bar to refine results (sorted by howmany results have the property) Clicking on one/filling out the field/setting a min max displays the results and sends out a new more specific query looking for those specific properties. New properties are obtained that way.

If I never see a pinterest link, or one of those sites that just republishes stackoverflow answers unedited, I'd be fine with it.

Of course, these systems always get abused and some political or news site will end up on it.

In other words, this is precisely how a market functions.

That's part of the point. There could be different search engines that run the same code, but have different sets of tier 1 domains that cater to different audiences. And if you have the resources, you could set up your own engine with a set of tier-1 domains that you chose.

It uses Gigablast which has a much more fair search result set more akin to search engines of the past!