The Swede behind search.marginalia.nu has had a working search engine running at a single desktop class computer in a living room, all programmed and maintained on his spare time, that was so good that in its niches (history, programming, open source comes to mind) it would often outshine Google.

Back before I found Kagi I used to use it everytime Google failed me.

So, yes, given he is the only one I know who manages this it isn't trivial.

But it clearly isn't impossible or that expensive either to run an index of the most useful and interesting parts of internet.

I think the problem with search is that while it's relatively doable to build something that is competitive in one or a few niches, Google's real sticking power is how broad their offering is.

Google search has seamless integration with maps, with commercial directories, with translation, with their browser, with youtube, etc.

Even though there's more than a few queries they leave something to desire, the breadth of queries they can answer is very difficult to approach.

This is one of those things that I think is interesting about how "normal" people use the Internet. I am guessing they just always start with google.

But for me, if I want to look up local restaurants, I go straight to Maps/Yelp/FourSquare(RIP). If I want to look up releases of a band, I go straight to musicbrainz. Info about Metal Band, straight to the Encyclopeadia Metallum. History/Facts, straight to Wikipedia. Recipes, straight to yummly. And so on. I rarely start my search with a general search engine.

And now with GPT, I doubt I even perform a single search on a general search engine (google, bind, DDG) even once a day.

"Normal" people don't start with Google, not any more. They start with Facebook, Instagram, X, Reddit, Discord, Substack etc. That's exactly the problem, the world-wide web has devolved back into a collection of walled gardens like things were in the BBS era, except now the boards are all run by a handful of Silicon Valley billionaires instead of random nerds in your hometown.

You have just described how "innovation" is often just a power shift, often one that does not benefit the user. Old is new again, but in different hands; the right hands, of course.

I've heard teenagers today often search first on tiktok. To find restaurants.

No search engine is refreshing every website every minute. Most websites don't update frequently, and if you poll them more than once every month, your crawler will get blocked incredibly fast.

The problem of being able to provide fresh results is best solved by having different tiers of indices, one for frequently updating content, and one for slowly updating content with a weekly or monthly cadence.

You can get a long way by driving he frequently updating index via RSS feeds and social media firehoses to provide singnals for when to fetch new URLs.

I meant this in response to the parent that Common Crawl only updates every month, which seemed to imply that this was sufficient.

This is too slow for a lot of the purposes people tend to use search engines for. I agree that you don't need to crawl everything every minute. My previous employer also crawled a large portion of the internet every month, but most of it didn't update between crawls.

Big fan of your work Viktor, thanks for everything you build and how much you document it

See also: IndexNow [1], a protocol used by Bing, Naver, Yandex, Seznam, and Yep where sites can ping one of these search engines when a page is updated and all others will be immediately notified. Unfortunately it does seem somewhat closed as to requirements for joining as a search engine.

[1]: https://www.indexnow.org/

What year was this thing created, because the /.well-known URI scheme[1] has existed for a long time and is designed for this kind of junk https://www.indexnow.org/documentation#:~:text=Hosting%20a%2...

1: https://www.iana.org/assignments/well-known-uris/well-known-...

It was announced October 18, 2021: https://blogs.bing.com/webmaster/october-2021/IndexNow-Insta...

The irony of a website from 2 major search engines looking like it was made in the early 2000s doesn't escape me. But, to my original point, there's absolutely no way they were ignorant of well-known URIs

Some sources update faster than others, you could index news sources hourly and low velocity sites weekly. Google does that. CommonCrawl gets 7TB/month, indexing and vectorizing that is quite manageable.

I think it is more complex than that. Common crawl does not index the whole web every month. So even if you use common crawl and just index it every month, which you could do pretty cheaply admittedly, I don't think that would lead to a good search index.

Running an index is an extremely profitable business, from multiple points of view (you can literally earn money, but also run ads, you get information you can sell, you can buy mindshare). Everybody is looking for indexes beyond Google and Bing, but there are none. If it really is as easy as indexing common crawl, then I think we'd have more indexes.

This reads like the “I could build Dropbox in a weekend”.

Haha yes, but my argument is not about individuals, but about "tech" companies that externalize everything and do not develop tech internally.

It's just files, how hard could it be?

No that hard, took me 4 weekends to build a private search engine with Common Crawl, Wikipedia and HN as a link authority source. Takes about a week to crunch the data on an old Lenovo workstation with 256gb ram and some storage.

For news-only there's https://littleberg.com