Back before I found Kagi I used to use it everytime Google failed me.
So, yes, given he is the only one I know who manages this it isn't trivial.
But it clearly isn't impossible or that expensive either to run an index of the most useful and interesting parts of internet.
Google search has seamless integration with maps, with commercial directories, with translation, with their browser, with youtube, etc.
Even though there's more than a few queries they leave something to desire, the breadth of queries they can answer is very difficult to approach.
But for me, if I want to look up local restaurants, I go straight to Maps/Yelp/FourSquare(RIP). If I want to look up releases of a band, I go straight to musicbrainz. Info about Metal Band, straight to the Encyclopeadia Metallum. History/Facts, straight to Wikipedia. Recipes, straight to yummly. And so on. I rarely start my search with a general search engine.
And now with GPT, I doubt I even perform a single search on a general search engine (google, bind, DDG) even once a day.
The problem of being able to provide fresh results is best solved by having different tiers of indices, one for frequently updating content, and one for slowly updating content with a weekly or monthly cadence.
You can get a long way by driving he frequently updating index via RSS feeds and social media firehoses to provide singnals for when to fetch new URLs.
This is too slow for a lot of the purposes people tend to use search engines for. I agree that you don't need to crawl everything every minute. My previous employer also crawled a large portion of the internet every month, but most of it didn't update between crawls.
1: https://www.iana.org/assignments/well-known-uris/well-known-...
Running an index is an extremely profitable business, from multiple points of view (you can literally earn money, but also run ads, you get information you can sell, you can buy mindshare). Everybody is looking for indexes beyond Google and Bing, but there are none. If it really is as easy as indexing common crawl, then I think we'd have more indexes.
When I search Kagi for "Hacker News", results start with this fine text:
65 relevant results in 1.09s. 47% unique Kagi results.
The first result is almost assuredly the right one, but either they're ruling out a lot of pages as not-what-you-meant, or their index is really small.
Kagi reduces mental load by default, and this is a good thing.
Having a low number of results has a net benefit of lower cognitive load for me, so I like how Kagi returns less results, not more.
But "more results" is counted towards a new search, and I didn't know that. Thanks for pointing out.
https://help.kagi.com/kagi/search-details/search-sources.htm...
They source results from lots of places including Google. One way that you can confirm this is to search for something that only appears in a recent Reddit post. Google has done a deal with Reddit that they're the only company allowed to index Reddit since the summer.
DuckDuckGo gets no answers if you specify only results from the last week: https://duckduckgo.com/?q=caleb+williams+site%253Areddit.com...
Kagi is fine if you do the same: https://kagi.com/search?q=caleb+williams+site%3Areddit.com&d...
edit: I don't think this is a bad thing for Kagi. I'm a very happy subscriber, and it's nice for me that I still get results from Reddit. They're very useful!
The only way there is a chance for me to afford Kagi might be to buy "search credit" without a subscription and without minimum consumption. And then it would only be good if they allowed more than 1000 domain rules and showed more results (when available)
It's great that they are developing their own index, but I'm skeptical that it makes up more than a tiny fraction of what they can get from Google/Bing. DDG has been making similar claims for years but are still heavily reliant on Bing.
This isn't to knock on upstart search engines. I think that Google Search has declined massively over the past 5-10 years and I rarely use it. More competition is sorely needed, but we should be be clear eyed about the landscape.
That way users get tailored search without losing scale.
Could you please elaborate?
Cliqz was the first time for me that a Google alternative actually worked really well - and it, or now brave search, is what parent was asking for :)
> We expect that advertising funded search engines will be inherently biased towards the advertisers and away from the needs of the consumers.
Brave Search Premium hasn't been around nearly as long as their free tier serving ads, and I'm not confident this conflict of interest is gone.
Having independent indexes is a win regardless though.
[0] https://snap.stanford.edu/class/cs224w-readings/Brin98Anatom...
Your facebooks/twiters/instagram/stack overflow/reddit ... And they all have limited expensive api's, and have bulk scrapping detection. Sure you can clobber together something that will work for a while, but you can't runn a buissness on that.
Aditionaly most paywalled sites (like news) explicitly whitlist google and bing, and if someone cretes new site, they do the same. As an upstart you would have to reach out to them to get them to whitelist you. and you would need to do it not only in USA but globaly.
Anothe problem is cloudflare and other cdns/web firewalls, so even trying to index mom and pops blog site could be problematic. An d most of the mom and pop blogs are nowdays on som ploging platform that is just another silo.
Now that i think about it, cloudflare might be in a good position to do it.
The AI hype and scraping for content to feed the models have increased dificulty for anyone new to start new index.
The decentralized nature of the internet was amazing for businesses, and monopolization could ruin the space and slow innovation down significantly.
The legal concept of fair usage has and is being challenged, and will best tested in court. Is the Golden Age of Fair Use Over? Maybe [0].
[0] https://blog.mojeek.com/2024/05/is-the-golden-age-of-fair-us...
If the site is less aggressively blocking but only has a per-IP rate limit, buy a subscription to one of those VPNs (it doesn't matter if they're "actually secure" or not - you can borrow their IP addresses either way). If the site is extremely aggressive, you can outsource to the slightly grey market for residential proxy services - for fifty cents to several dollars per gigabyte, so make sure that fits in your business plan.
There's an upper bound to a website's aggressiveness in blocking, before they lose all their users, which tops out below how aggressive you can be in buying a whole bunch of SIM cards, pointing a directional antenna at McDonald's, or staying a night at every hotel in the area to learn their wi-fi passwords.
I am familiar with most of that, and there is a BIG difference between trying to find a workaround for one site, that you scrape ocasionaly, than to to find workaround for all of the sites.
Big sites will definitely put entire ISP's behind annoying capachas that are designed to stop exactly this (if you ever wonder why you sometimes get capatchas that seem slow to load, have long animations, or other annoying slow things, that is why etc.)
And once you start making enough money to employ all the people you need for doing that consistently, they will find a jurisdiction or 3 where they can sue you.
Also good luck finding residential/mobile ISP's that will stand by, and not try to throttle you after a while.
You definitively can get away with doing all of that for a while, but you absolutely can't build sustainable businesses on that.
Those example 3 bullet points of today's improved 2024 computing power you list isn't even enough to process Google's scale 14 years ago in 2010 when the search index was 100+ petabytes: https://googleblog.blogspot.com/2010/06/our-new-search-index...
A modest homegrown tech stack of 2024 can maybe compete with a smaller Google circa ~1998 but that thought experiment is handicapping Google's current state-of-the-art. Instead, we have to compare OSS-today vs Google-today. There's still a big delta gap between the 2024 OSS tech stack and Google's internal 2024 tech stack.
E.g. for all the billions Microsoft spent on Bing, there are still some queries that I noticed Google was better at. Google found more pages of obscure people I was researching (obituaries, etc). But Bing had the edge when I was looking up various court cases with docket #s. The internet is now so big that even billion dollar search engines can't get to all of it. Each has blindspots. I have to use both search engines every single day.
Most of Google's 100PB is picture and video. Filtering the spam and deduping the content helped Google reduce the ~50B page index in 2012 to ~<10B today.
I'm sure all the LLM providers are already considering this, but there's so much important information that is locked away in videos and pictures that isn't even obvious from a transcript or description.
> But what if I don't want to search Reddit, stack overflow, and blogs from the early 2000s
That is a strawman. There are huge numbers of websites (including authoritative ones like governments and universities) and a lot of content.
> There is an entire working generation that never heard a modem sound and has never even made a consideration for making sure their content is accessible in plaintext.
If they want video they will do the same as everyone else and search Youtube. Different niche.
> I'm sure all the LLM providers are already considering this, but there's so much important information that is locked away in videos and pictures that isn't even obvious from a transcript or description.
That is true, but if you are getting bad search results (and the market for other search engines are people who are not happy with Google and Bing results) that does not help much are you are not seeing the information you want anyway.
Ya know... a search engine that was limited to *.gov, *.edu and country equivalents (*.ac.uk, etc) would actually be pretty useful. Ok, I know you can do something like it with site: modifiers in the search, but if you know from the beginning you're never going to search the commercial internet you can bake that assumption into the design of your search engine in interesting ways.
And the spam problem goes away.
Hmm.
There is no open web anymore. Google killed it. There are probably fewer than 100k useful websites in the world now. Which is good for startups, because the problem is entirely tractable.
No matter what type of market analysis I do, I almost invariably find there's something different that say, the Koreans or the Europeans are using. The Yelp of Japan is Tabelog, the Ubereats of the UK is deliveroo, the facebook of russia is vk.ru etc.
That's really the beach head to capture - figure out what a "web region" is for a number of query use-cases and break in there.
If you went into Blockbusters then there was actually a small subset of the available videos to rent. Films that had been around for decades were not on the shelves yet garbage released very recently would be there in abundance. If you had an interest in film and, say, wanted to watch everything by Alfred Hitchcock, there would not be a copy of 'The Birds' there for you.
Or another analogy would be a big toy shop. If you grew up in a small town then the toy shop would not stock every LEGO set. You would expect the big toy shop in the big city would have the whole range, but, if you went there, you would just find what the small toy shop had but piled high, the full range still not available.
Record shops were the worst for this. The promise of Virgin Megastore and their like was always a bit of a let down with the local, independently owned record shop somehow having more product than the massive record shop.
Google is a bit like this with information. Youtube is even worse. I cottoned on to this with some testing on other people's devices. Not having Apple products, I wanted to test on old iPads, Macbooks and phones. For this I needed a little bit of help from neighbours and relatives. I already knew I had a bug to workaround, and that there was a tutorial on Youtube I needed to do a quick fix so I could test everything else. So this meant I had to open Youtube on different devices owned by very different people, with their logged in account.
I was very surprised to see that we all had very similar recommendations to what I could expect. I thought the elderly lady downstairs and my sister would have very different recommendations to myself, but they did not. I am sure the adverts would have been different, but I was only there to find a particular tutorial and not be nosy.
I am sure that Google have all this stuff cached at the 'edge', wherever the local copper meets the fibre optic. It is a model a bit like Blockbusters, but where you can get anything on special request, much like how you can order a book from a library for them to get it out of storage for you.
The logical conclusion of this is to have Google text search becoming more like an encyclopedia and dictionary of old, where 90% of what you want can be looked up in a relatively small body of words. I am okay with this, but I still want the special requests. There was merit in old-school Alta Vista searches where you could do what amounts to database queries with logical 'and's 'or's and the like.
The web was written in a very unstructured way, with WYSIWYG being the starting point, with nobody using content sectioning elements to scope headings to words. This mess suits Google as they can gatekeep search, since you need them to navigate a 'sea of divs'.
Really a nation such as France with a language to keep need to make information a public good with content structured and information indexed as a government priority. This immediately screams 'big brother', but it does not have to be like that. Google are not there to serve the customer, they only care about profits. They are not the defenders of democracy and free speech.
If a country such as France or even a country such as Sweden gets their act together and indexes their stuff in their language as a public good, they can export that knowhow to other language groups. It is ludicrous that we are leaving this up to the free market.
If you leave it up to the government, inevitably you're going to get only information approved by the people in power in that government.
You could call that search engine "Pravda".
(I don't think the open web is dead, but it's looking awfully unwell).
Excerpt from the article: >[...] Caffeine takes up nearly 100 million gigabytes of storage in one database and adds new information at a rate of hundreds of thousands of gigabytes per day. [...]
Your comment did make me pause and sanity check the math: https://www.google.com/search?q=%22100+million+gigabytes%22
In any case, a lot of people translated "100 million gigabytes" to "100 petabytes" based on that blog : https://www.google.com/search?q=google+search+index+estimate...
What's the current best estimate of its size now in 2024?
Another major factor is that building a search index and algorithms that searches across billions of pages with good enough latency is very hard. Easy enough for 10s of millions scale search but a different challenge for billions.
Some claim(ed) click-query data is needed at scale, and are hoping for that remedy. Our take is what is the point of replicating Google. Anyway, will this data be free or low cost? You know the answer.
Cloud infrastructure is very expensive. We save massively on costs by building our own servers, but that means capital outlay.
Are any individual users downloading CC for potential use in the future?
It may seem like a non-trivial task to process ~100TB at home today but in the future processing this amount of data will likely seem trivial. CC data is available for download to anyone but, to me, it appears only so-called "tech" companies and "researchers" are grabbing a copy.
Many years ago I began storing copies of the publicly-available com. and net. zonefiles from Verisign. At the time it was infeasible for me to try to serve multi-GB zonefiles on the local network at home. Today, it's feasible. And in the future it will be even easier.
NB. I am not employed by a so-called "tech" company. I store this data for personal, non-commercial use.
Finding some hack to democratize&decentralize the indexing and expensive processes like JavaScript interpretation, image interpretation, OCR, etc is an open angle and even an avenue for "Web3" to offload the cost. But you will ultimately want the core search and index on a tighter knit cluster (many computers physically close to one another for speed of light reasons, although you can have N of these clusters) for performance so it's a hard nut to crack for making something equitable for both the developers and any prospectors and safe from various takeovers and enshitifies. Let us know if you know a way.
Thanks to all the SPA idiocy you will miss enough content to matter if you have zero JS interpretation, so you would want to let the user choose which indexes they want for a query because sometimes you need these other resource types to answer the query.
Moreover, get out of your echo chamber and you’ll see that for a majority of humanity Google is the internet. You have to supplant the utility, not just the brand. Most businesses cannot handle the latter let alone the former. If you want something to replace Google, you have to think about replacing the internet itself. But not many are that bold.
why do you think amazon's trying to lay people off? they want them to create new startups that they can acquire.
1. Create your own index/crawl and true search engine on top of it, rather than delegating that back to bing or Google. In doing so, solve or address all the related problems like SEO/spam, and cover the “long tail” (expensive part)
2. Monetize it somehow to afford all of the personnel/hardware/software costs. Probably with your own version of Google’s ad products.
3. Solve user acquisition. Note that Google’s user acquisition process involves multiB/yr contracts with Apple and huge deals with other vendors who control the “top of funnel” like Mozilla Firefox, and that this is the sole purpose of chrome/android/chromebook/etc who you’ll never be able to make a deal with. You will probably at the very minimum need to implement your own platform (device, OS, browser alone probably won’t cut it).
4. Solve the bootstrap problem of getting enough users for people to care about letting you index their site, without initially being able to index a lot of sites, because you are not important enough.
5. Somehow pull all of this off in plain site of Google (this would take many years to build both technically and in terms of users) without them being able to properly fend you off
6. Somehow pull all of this off in spite of the web/web search seeming like it’s going to die off or fade into irrelevance
OR you can dedicate a decade of your life to something more likely to succeed.
There is one way to compete with Google search. Google search is a general search engine. But what if you only wanted medical information? information about cars? information about physics? electronics? history? etc.
Specializing makes for a much smaller scope of the problem, and being specialized means it could deliver more useful results.
For example, imdb.com. I don't ask google about movie stuff, I go to imdb.com because it specializes in movie info.
Some of them charge a lot for access but they certainly exist.
One, the easiest way for them to fight back is to buy you. Two, you're writing off the Innovator's Dilemma that has put them into this mess. Correcting course means giving up a lot of ad revenue and, with it, market cap.
You run this bloated mass as a co-engine to your search. If you stumble upon any of the digested sites or articles you just have the site-eater blob regurgitate a html re-creation locally. They get no traffic, you get whatever they wrote purged of all links and with optional formatting to remove the fluff BS copywrite that many of these sites pad the tiny core of usefulness with.
A solution could be more than one algorithm being used to rank results, i.e. other engines, other rules. They'll likely use many of the signals available to them that Google uses for quality and relevance, but highly unlikely a genuine alternative search would rank them exactly the same- and much more unlikely an SEO could rank well in multiple engines.
The aff links aren't the problem, it's the proliferation of pages that are created solely to rank and get the links clicked on. Sometimes the content is useful, sometimes it's padded nonsense.
What do you mean? Google gained dominance just being a dot com URL. This in a time when competitors were already very well established.
Google's dominance will never be upended by an incremental improvement.
Only people I saw manually typing the url are my office colleagues(mostly engineers of some sort), even management does the same(skipping the FB part obviously) and you can see it when they are sharing their screen and they want to search for something.
To pull people out of google moat, first step is becoming their default search engine.
Eg see how chatgpt grew
Even then after using Google Gemini subscription the last few months I think the problem is not Google search rather the web ecosystem as if google gives me most of the answers without having to click any link you and I might be happy but billion people living off those links directly or indirectly won't be happy.
Here's one list:
>A look at search engines with their own indexes
https://seirdy.one/posts/2021/03/10/search-engines-with-own-...
HN discussions:
https://news.ycombinator.com/item?id=26429942
The only viable option when starting small is to grow in the cracks of Google with features that users really love that Google won’t provide.
I understand that competing with Google was borderline impossible a decade ago. But in 2024, we have cheap compute, great OSS distributed DBs, powerful new vector search tech. Amateur search engines like Marginalia even run on consumer hardware. CommonCrawl text-only is ~100TB, and can fit on my home server.
Why is no company building their own search engine from scratch?