The keyword(no pun intended) being was --- I remember going through many pages of relevant results and finding the "needle in a haystack" many times with Google, but now it seems like anything even vaguely obscure gets absolute rubbish results (and it claims there are no more results if you try to go deeper, despite the fact that I know the pages are out there.) I'd say it started to decline roughly 10 years ago.

Google has definitely just plain delisted a lot of older sites, even if I do exact quote matches I can’t get results for sites I know exist and used to be indexed and often turn up on other search engines.

On top of that, as you say, anything that isn’t from a huge web property or corporate site is so heavily penalized you’re lucky to ever see it even if it is new.

The utility of google as a web search engine is definitely declining over time.

This is possibly a self-destructive pattern, too. Once nobody can find anything outside of Quora/Wikipedia/Pinterest/Stackoverflow etc. why would anyone bother creating small sites/blogs (this has been happening already for a decade) — But then, once the entire accessible web is just these handful of big sites, what is Google’s utility? They all have internal search and are increasingly accessed primarily through apps

Like grocery stores or many types of restaurants, for example. They usually can't compete against the Walmarts and Paneras, which suck up consumer mindshare. Similar with soft drinks. Coke and Pepsi build as much mindshare as possible. You may invent a great product, but it likely will never get noticed among corporate noise.

Google, Wikipedia, Facebook are becoming the default go-to spots for particular activities, similar to what McDonald's did to fast dining in the 60s and 70s. I guess, in general, markets saturate, oligopolize, and the small spots fade away unless they're niche or very high quality.

It's even delisting a lot of content from existing sites. For example, I've got previous HN comments that I used to reference using `akiselev site:ycombinator.com [some key words]` that are nowhere to be found now. I started noticing this a few years ago.

So much that it could disappear and we would be able to move on. Hard to say that a few years ago.

>Google has definitely just plain delisted a lot of older sites

This really sucks. What are the alternatives?

> It's hard to explain just how good Google was back in the day, compared to those early search engines.

This 2000 New Yorker article does a pretty good job:

https://www.newyorker.com/magazine/2000/05/29/search-and-dep...

It was actually how I discovered Google. I was a grad student at the time (English literature) and dreamed of accessing all the primary and secondary sources on a particular topic from the convenience of my own keyboard (still a pipe dream today). I cycled through every search engine I could find looking for one that would actually return results relevant to my search terms. I remember AltaVista being touted as one of the best but still failing to satisfy.

As soon as I read this New Yorker article, I checked out google.com and had that same eureka experience: finally, this is the one.

In grad school, in the early days of the dotcom boom, some of us CS-ish people curious about startups would talk about them. One day, one of the PhD students asked everyone which dotcom's stock would they most like to have. I said Google. (It might not have been a company yet, but we kinda assumed.)

Google worked surprisingly well, obviously they were going to be hugely important (unless another player appeared and did a big next leap), and obviously they were smart. (On the side, I also got warm-fuzzies that the search hits I got seemed to have a strong Linux bias.)

The grad student who asked, coincidentally, ended up deciding not to finish his PhD, and went to Google. :)

I remember being blown away by how good google was around those times.

But I Waldo remember just how blown away I was when I first saw altavista do it’s thing. Altavista was as equally amazing in 1995 as google was in 2000. Innovation goes in waves.

I donno, for me early google was horrible. I always went back to AV because it gave me the results I was looking for. But over time AV just got slower and bloated and Google just got better.

Glad to see someone else had the same experience. When someone recommended Google to me, I tried it and immediately went back to AltaVista. The only distinctive characteristic I remember noticing was the immense whitespace on the Google home page. That sort of simplicity in page design was different from other search and portal websites at the time.

Using AltaVista for me meant digging through page after page of results. Comprehensiveness was the main concern. It was up to the user to evaluate the relevance of the results.

Early Google made similar claims about comprehensively searching millions of pages, but as we know today, they are intent on inferring meaning and purpose. They actively discourage and prevent users from combing through page after page of results. User evaluation (i.e., intelligence) is not expected. Google attempts to evaluate results for the user based on popularity, originally estimated primarily by counting backlinks. Popularity as a filter is useful sometimes but deeply flawed at others. It's arguable Google has dulled, atrophied or stunted development of web users' analytical skills. When it first appeared on the web, Google had no paid placements and no advertising. What was not to like? They later abandoned their original mission to avoid the influence of paid placement. They became beholden to advertising.

It was not difficult to see when and where the influence of advertising came into AltaVista. However when this started to happen at Google, Google tried to hide the ads by making them text-only. As if the influence was not there.

We need another AltaVista, where user evaluation of results is allowed and encouraged, with a mission statement like the original Page and Brin paper announcing Google: no influence by advertising. Ultimately PageRank was dependent on human discretion: the decision whether or not to link to another page. We soon learned that this discretion, this choice to link or not to link, easily becomes driven by money when people know it effects PageRank. Google quickly got gamed and it has been trying to pretend it can manage this ever since.

We need non-commercial search.

I also had this experience. However, I now feel all search engines are returning worse results. Very rarely can I find anything meaningful outside of major content providers. Everywhere my results are filtered through a local government lens. In many ways it feels like search on Facebook, YouTube, Twitter, Instagram, LinkedIn, WeChat, closed appstores, etc. have become the new defacto content aggregation mechanisms, and those are tending toward feeds and curated content rather than requiring active search. The best and most meaningful results on most actual search engine results pages are Wikipedia excepts, or trivial widgets (currency or unit conversion results, weather forecasts, etc.). I don't have any objective measures and am unlikely to be a representative sample but I feel I am searching less, my browser has become a URL-bar based memory system, the traditional 'home page' notion has been replaced by a Firefox auto-curated most-frequent-sites list, and I use traditional search engines more like a command line than a library index. OTOH we now have Library Genesis which is brilliant.

Irrespective of its legal status, LG is proof one does not need 75,000 employees, nor any perceived "brilliance", to provide a world-class, non-commercial information retrieval service. It can hold its own next to many university libraries' remote access facilities and it puts Google Scholar to shame. It is command-line friendly and offers bulk data. Unlike Google, LG does not try to guess what the user is searching for and answer questions. One encourages traditional learning, the other is incessantly trying surveil its users in the name of selling online ad services to third parties.

does libgen provide full text search. I thought it only provided title/author search.

IIRC, the draw was "you don't have to use plus signs" and the other options, and you didn't...until Google started trying to read between the lines of our searches and we started wishing Google had plus signs.

Google's aversion to zero-result queries is a problem, and possibly the problem, with their "accuracy," so to speak.

I remember the first time I had heard of Google -- it was ALREADY used as a verb!

I was interning at Xerox PARC (adjacent to Stanford campus) in summer 2000. As far as I remember, I used Alta Vista at the time.

Somebody asked a question, and one of the researchers/intern mentors said to "Google it"! I think there were some puzzled looks, but we tried it, and I started using Google and never went back.

When I returned to college in the fall, I remember my former housemate saying how good Google was too. She had started using it too. Once people started using it, they never stopped!

Agreed. I was a heavy user of Info Seek at the time (I especially liked how they highlighted search terms in the results) but it was hard to deny how much better Google results were. For a long time I just used Info Seek first and would try Google second, but after a while I realized I was just using Google. The results were impressive.

I used to go to library to find CS books samples and read them for about 1-2 weeks before I actually get to code. After a classmate share Google with us it was just unbelievable the quality of results and how easy was for us to start finding sample code and literature. Yes, the best tool probably in modern human history

The main thing I remember about Google was that it was fast, really fast compared to AltaVista, which was becoming so slow as to be nearly unusable (sometimes you’d wait 30 seconds for a page load).

Weirdly similar experience. I too discovered Google in my college CS lab, when I noticed the (then) weird looking homepage. I had been swearing by AltaVista at the time. But it was with Google that I was able to, for the first time, get what I wanted without having to go through multiple pages of search results. A search engine giving you what you want within the first few results of your very first search attempt was just not a thing. Crafting search queries was an art, and so was wading through results.

On the other hand, search engines of those times gave you what you asked for, not what it decided was best for you. That option is gone now, and searching for many niche topics is no longer possible. If that's what you're looking for, it might as well be 1990.

I wonder if the "I'm feeling lucky" has stayed around because it's a good training heuristic for Google Home (and maybe other services?) even if very few people actually use it. I'm genuinely curious why they've kept it around.

My guess would be it’s more of a branding exercise. Quirky, colourful, friendly.

It's also pretty ideal feature to keep around even if it is mostly unused.

1) It isn't very complicated -- it doesn't require a lot of code that is going to rot.

2) When it is used, it's cheaper rather than more expensive.

3) It only "clutters" a UI that isn't actually seen that often nowadays -- you have to actually go to google.com, rather than just typing in the browser bar -- and isn't particularly cluttered.

However, if it makes the experience worse then it’s not worth keeping. I’m sure they did some study where they found that most users weren’t happy and had to go back.