It does exist, just the same - probably more so, because the web as a whole is many orders of magnitude large now, so there's just more of everything.

However, 90% of everything is crap, so the good stuff is not buried under thousands, or millions, or mere billions of pages of crap, but trillions.

> any tom, dick and harry could open up note pad and get a website online.

This part hasn't changed one iota - you can still do _exactly_ that, even following a tutorial from 1996 to the letter, if you want. All that stuff still works fine - and I have websites that old, built that way, that are still up and work fine.

> It's my opinion that those who complain about Google search results are frustrated that Google can no longer find a web that no longer exists

You ever try to search for a niche piece of software for your computer? Google surfaces the "Top 7 Best software" content farm articles before you ever get to the official sites. That's what some of us are frustrated about.

Have you ever searched for a recipe in Google? Have you noticed that the top results always go on for a few pages of SEO-optimized storytelling and explanation before you actually see the recipe?

I'm sure someone at Google has. They could fix it if they wanted to, and boost well-known sites that get straight to the recipe (allrecipes.com, foodnetwork.com) instead of making you scroll through the SEO spam first. They haven't.

I used to think this too but if all the recipe websites start going straight to the recipe without the the personalization, then how would you rank them? Remember, there are thousands of these websites while there are only so many ways you can boil an egg, for example.

PageRank-like system, ideally based on real identities? (i.e. depending on how often the recipe is linked from Facebook, etc.)

Also if you are specifically doing it for recipes, just select manually websites that have precise ingredient amounts, very detailed and explicit recipes, that explain the reasoning behind each step, provide photos of how things should and shouldn't look like, why the proposed technique is optimal and/or the tradeoff of alternative techniques, the chemistry involved, comparison with other recipes, etc. (although in my experience most or all recipe sites are horrible and aren't even vaguely close to something like this)

It is largely Google's fault.

Example. If you searched the word "flight" in 2002, the first two results would be NASA links and the third would be the Museum of Flight. These pages still exist. But now the entire first page of Google search results are cost comparison websites. This is Google's badness, not the web's.

I mean, you do you but when I search flight I for one am looking to purchase a flight. If I want NASA links or the Museum of Flight I would specify that.

Alternatively, I could have searched for "flight comparisons" or "US airlines" back in 2002 if that's the type of result that I actually wanted.

The problem is that there is no way I can tell current Google to give me stuff that's interesting, novel and information dense anymore. It's all well and good if I knew that the Museum of Flight existed before my search, but the point of Google is to tell me about its existence. I can't tell Google "flight ... but no cost comparison or airliner website stuff"; such a query is currently not possible.

It's not possible because the demand for this is very low and implementation cost is extremely high. No one will pay for that.

> demand for this is very low and implementation cost is extremely high.

Demand is real as evidenced by everyone in this thread wanting such a search engine. We are not representative of genpop, but we aren't insignificant either.

Google had this in 2002 (i.e. a default to novel and info dense results unless otherwise specified), so it can't be that hard or expensive.

> No one will pay for that.

So we have ads.

> They haven't given up; the OP has a point. The "sites" you are hoping for Google to return _don't exist_.

I was beginning to wonder if this explanation was true but after starting to use Marginalia (mostly for fun) I now know the old web still exists - Google just doesn't let me see it.

They have given up, as in if you Google anything technical these days you'll get tons of spam sites who just copies the content from GitHub and StackOverflow. Google used to punish sites that just copied content from other sites, but that isn't the case anymore.

No. There still are forums on lots of topics. But search how to fix X and instead of prioritizing a forum site with answers you get 5 top results trying to sell you the parts to fix X. The answers are always way down the list.

>There still are forums on lots of topics.

What is Reddit if not the largest forum in the world? You have the same gripe; you are looking for a web where people share things for free (like in forum); and when Google first existed most of the web is like that.

And if google were to prioritize forums; then SEO blackhats would do everything in their power to make every website look like a forum; and then you are back to square one. The problem is the profit motive, not Google's algorithms.

Yes but google crawls the internet it would be pretty trivial for it to scan a site and realize it was full of affiliate links and trying to sell things. They could rank those sites lower then just a site with text and not trying to sell everything.

Explain away how literally verbatim searching with quotes doesn't return verbatim results. Or how searches that match the very title of an article returns mountains of SEO spam before listing the article, if it shows up at all.

If there's an algorithm being gamed that causes that, that algorithm was designed explicitly to be gamed.

Google intentionally finds synonyms for obscure jargon that would have typically turned up these sites. The verbatim setting barely makes a difference.

The reliance on synonyms and semantically related topics has made iterative refining impossible as well, it's essentially non-converging.

My solution has been to search in multiple foreign languages, which looks less conducive to these "affordances".

Yeah, having spent a lot of time working with dense vector based techniques, especially in the context of a domain which is really similar to semantic search, I can say 100% that I don't want BERT and related techniques anywhere near my google search. That's the real crux of the problem, is the reliance on these sorts of models, and their utilization within query rewriting schemes.

YES exactly THIS.

When the "No Search results" page was deleted, the internet died.

Sure they do. It is just in 2001, there was 30M websites, and let's say 5% of them were good. And in 2020, there is 2000M websites... and the number of good ones is about the same (because there are only so many people willing to spend time making personal website). So the fraction is much lower now.

It is easier than ever to open your websites, and there are tons of free providers today. Look at any manual aggregator -- like hacker news or hackaday.com -- to see them. This is a variety I have not had in 2001!

Part of the problem is that Google started heavily favoring the age of a domain about a decade or so ago. If you spin up a new site, with high quality content, you simply won’t show up in Google for a long time.

Also, the amount of good content today is vastly larger than it was in 2001. Though you are right about the ratio changing.

Back in 2001 advertising was far more lucrative than it was today. So a website could be both: useful, and a for-profit, ad-supported enterprise. Hell, the reason why most online versions of newspapers were free was because advertising was so lucrative. Now, ads pay peanuts, and only make sense if you already have ludicrous scale, or are able to game search results with low-effort """content""" to get that scale.[0]

So it's both "less signal" (fewer high-quality websites) and "more noise" (more spammy SEO-to-oblivion sites).

[0] A similar thing happened to video advertising, which is why even really good YouTube videos are filled to the brim with ad integrations, merch tie-ins, and calls to like and subscribe.

In this vein it used to be easy to find dissenting opinions on something:

<Topic> sucks

RIP "sucks".

Because its simply not possible. There are too many barriers now. You need to get a static ip for one. Domains cost more. Email for those domains costs quite a bit. And your friends are far less likely to visit since theyre all addicted to infinite scrolling websites

> It's my opinion that those who complain about Google search results are frustrated that Google can no longer find a web that no longer exists

That, and the Google search results are bad for the parts that do exist.

Well you are stating your opinion as fact, which is not useful when your opinion is wrong, which it is in this case.

Still lots of great content still out there, often times the target of my original Google query.