Where are these figures coming from?

But what if I don't want to search Reddit, stack overflow, and blogs from the early 2000s and all the content you just threw away as irrelevant actually contains information I am looking for. There is an entire working generation that never heard a modem sound and has never even made a consideration for making sure their content is accessible in plaintext.

I'm sure all the LLM providers are already considering this, but there's so much important information that is locked away in videos and pictures that isn't even obvious from a transcript or description.

There is still large opportunity. Most of my searches are for plain text information.

> But what if I don't want to search Reddit, stack overflow, and blogs from the early 2000s

That is a strawman. There are huge numbers of websites (including authoritative ones like governments and universities) and a lot of content.

> There is an entire working generation that never heard a modem sound and has never even made a consideration for making sure their content is accessible in plaintext.

If they want video they will do the same as everyone else and search Youtube. Different niche.

> I'm sure all the LLM providers are already considering this, but there's so much important information that is locked away in videos and pictures that isn't even obvious from a transcript or description.

That is true, but if you are getting bad search results (and the market for other search engines are people who are not happy with Google and Bing results) that does not help much are you are not seeing the information you want anyway.

> That is a strawman. There are huge numbers of websites (including authoritative ones like governments and universities) and a lot of content.

Ya know... a search engine that was limited to *.gov, *.edu and country equivalents (*.ac.uk, etc) would actually be pretty useful. Ok, I know you can do something like it with site: modifiers in the search, but if you know from the beginning you're never going to search the commercial internet you can bake that assumption into the design of your search engine in interesting ways.

And the spam problem goes away.

Hmm.

That explains why you were 10 times more likely to find something 15-20 years ago then you are today. They reduced the size by dropping a lot of sites and not crawling as much. We expect google to be at 100PB x 100 with the growth of users and content over that time period. Someone made the decision to prioritize a smaller size over a more complete index and some A/B test was run and turned out well.