This is what everyone thought in 1999. Search engines were a solved problem in a crowded market where there could be little room for new contender And whatever limitations we had in search were just inherent in how it all worked and we just had to deal with it.

We must have lived in different 1999's

But that's not where others failed. Instead, it's coming up with the idea, realizing that it's a truly good idea, that was the important thing. (Remember, Yahoo and other(s) didn't want to buy Google and the algorithm, when they had the chance.)

Maybe if you replace "totally low hanging fruit", with "It was undiscovered winner-takes-it-all markets" I'd agree.

In hindsight, everything is obvious. But initially, discovering that opportunity — looking at how many people (how few, just 2) did, it wasn't easy.

I don't understand the distinction you're trying to make. I think what's being proposed is that the more undiscovered markets get discovered, the fewer undiscovered markets are left, and the more difficult they are to discover. Pagerank was a good idea that worked.

And back at the time, before Google Search, many many related markets didn't, couldn't, yet exist. And, back then, I'd think PageRank could count as one of the few and difficult thing to discover.

I think "low hanging fruit" makes it sound too easy :- )

Then you missed the point. It doesn't matter whether PageRank was something trivial any grad student could come up with or if it was on the level of Einstein's General Relativity in difficulty. The point is that it was one basic idea.

Today if you want to dethrone Google you need to overcome the enormous amount of engineering that has gone into Google Search and Maps. You can't do that by just "discovering" something as a grad student. It's going to take thousands (or millions) of engineering hours to achieve.

That is what it means for the low-hanging fruit to be gone. It's like the difference between discovering electricity, as we all know took quite a while but was achieved by a small number of scientists and inventors over a period of a couple centuries, and trying to compete against the modern-day electrical distribution network on your own, which is essentially impossible without some kind of Star Trek alien galactic empire level technology.

The big guys at that time were blinded by their own success, Google wanted users to do their search and leave the site which was completely opposite to what the Yahoo's of that day wanted to do, they wanted users to hang out on their portal.

I’ve often heard the saying how “everything easy has already been invented” and that it’s so much harder today to invent new things.

I think it’s a fallacy. Things were just as hard in the past.

I owned an ISP in 1998 and there were plenty of search engines at the time. Google invented something unique and innovative and they were rewarded tremendously for it. There were literally hundreds of other companies trying to do the same thing but Google was better. If it was low hanging fruit then Hotbot or any of the other major players could’ve done it.

In 1998 there were plenty of search engines and their use was distributed more evenly. The internet was still fairly new and most people weren't online yet. It certainly wasn't an integral part of most people's life. Google came along and improved on the existing search engines and in doing so wiped out a lot of that competition, becoming what they are today.

A new search engine now has to compete with global network effects, orders of magnitudes more data, Android, and a complex web of interconnected functionality.

Even only looking at the advantage that Google Maps brings to the table totally blows past any barriers present in 1998. Sure, you could build a much better search engine but who's going to use it when it can't give them directions?

Then consider that if you look like you'll end up making some headway, you're likely to just get bought and killed off/integrated.

Are you a pessimist in life? You’re focusing on only the negative aspects of starting a search engine today.

24 years after Google was formed we have orders of magnitude cheaper processing, orders of magnitude better AI, the ability to start small with cloud computing and work your way up from there. We have 24 years of search engine research to a large extent publicly available.

I really think all this “things would’ve been so much easier back then” are simply excuses as to why someone can’t do something today.

They do exist, have you heard of any of them?

My intention wasn't to dismiss the possibility of popular new search engines. Only to highlight that the environment is substantially more complex/challenging than in 1997.

I actually think that their entrepreneur spirit and business attitude made them successful. Other guys thought of their search engines as of technical experiments and hobbies, they weren't serious about it. In another words Google cared more for innovating their search engine, making money and then reinvesting it back in R&D and staying ahead of everybody. The same story was with Microsoft and Digital Research; Bill Gates simply cared more business wise and was more fanatical in making money than Gary Kildall.

I'm not a programmer but I can write down on a piece of paper an algorithm which is more efficient than Google's in filtering out "spam" websites. I'm from southeast Europe and for my local market there are numerous ecommerce phishing websites that are popping up on a first page of search results. Some even ranked first. I reported them to Google but 6 months after nothing changed.