I did just that. Well not specifically aimed at research, but the types of websites I'm interested in, written by humans, blogs and such. It's got some quality issues right now because I fucked up some of the keyword extraction logic and it's taken a while to identify just what's gone wrong.

Even though it's a bit broken, it has some lucid moments. Just compare:

https://www.google.com/search?q=mechanical+keyboards

https://search.marginalia.nu/search?query=mechanical+keyboar...

The take-away I want to drive home is that it's absolutely possible to build something the scale of Google c.a. 2003 and run it on consumer hardware. I think, due to general difficulties in making these things profitable, the ideal approach is to make the operation so absurdly cheap it can be run non-profit instead. I'll gladly pay out of my own pocket to have a good search alternative.

I love Marginalia, it's such a great project. You get really weird, but interesting results. Honestly I don't use it as much as I'd like, but it's still fun to go down a rabbit hole and read actual content.

It's probably quite far off being viable as a daily driver search engine, but maybe it doesn't need to be either. I built it to pick up the slack in the areas where the big search engines are floundering, and as long as it's doing that, I think that's quite sufficient.

Thanks for your work! FYI I've submitted a !Bang on DuckDuckGo to search Marginalia (see: https://duckduckgo.com/bang). Waiting for it to be approved.

I think it's been submitted a few times before, but I guess if they keep getting these requests they'll eventually add it.

Your project makes my life better both because it is already useful and because it gives me hope!

"random websites" is a great rabbit whole!

I want a curated search engine so badly. I feel like some enterprising developer could craft one specifically geared towards developers. Imagine a search that doesn't ignore operators like '?.' and where you can set persistent conditions on every single query without having to type it out. This would help prevent you from needing to type in the language you are searching for in every query.

Oh, and the cherry on top is completely abandoning the idea of Natural Language Processing. Go right back to keywords only.

I've considered taking a stab at it, but I didn't for likely the same reason no one else has: no idea how to monetize it. Sure you can run ads, but your conversion rates will likely stink because you'd be selecting for non-suckers.

Not trying to spam but will mention devonagent once more in this thread, they just sell it, old fashioned perpetual license. respects boolean operators.

https://www.devontechnologies.com/apps/devonagent#editions

Thanks for the recommendation. Just trying it out and it is much better than doing a direct search on google

You might be able to monetize through closer partnerships with companies like Microsoft, Atlassian, Amazon, etc.. I'm envisioning an ad system that is basically geared towards selling their cloud and DevOps services. Because the engine is geared towards programming queries, you might also entice these companies by showing that developers have an easier time troubleshooting their systems. Perhaps, eventually, the engine could be bought out and act like a loss-leader or funnel to services. I know Microsoft is trying to be very developer friendly.

Just some random thoughts. Monetization is a huge bootstrapping challenge for something like this.

I’d love to take a stab at it. I think there are ways to monetize it that don’t involve advertising (at least enough to cover salaries and infrastructure). The only reason I haven’t is I don’t have access to the initial capital.

Subscriptions. You would have to take money directly.

It can be solved without centralized server: every content site, page, or paper, will publish Bloom filter for their content discoverable via sitemap, to which you subscribe via RSS/Atom. When you need to search, you will make hashes for words in your query, then will check each Bloom filter for potential matches. When potential match is found, full page source can be downloaded and checked more carefully for ranking.

Perhaps technical users are more likely to be running an ad blocker and/or are overall less likely to actually click on ads. If you're going to have 1/10th the CPM by making this search engine vs. one geared towards everyone, why would you spend the extra time and CPU cycles powering it? Maybe a subscription model would work but you'd have to show just how better it is since you're competing against good-enough free search engines.

Content curation doesn't scale, so I believe it's out of question for the modern web. The best you can hope for is to have someone throw some quality AI at it, plus a lot of manual tweaking.

Given that the goal is only a small subsection of topics, might scalability not be as critical to the success of the project? If the engine is specifically crawling official programming language documentation, help forums like StackOverflow, and select sites/pages (think university professor with resources about how to implement xyz), might that be enough to not necessitate scaling even farther?

Not only that, we just need one that works the way Google used to work. You know, something like the old algorithm that made Google famous and made competitors obsolete.

The hard part of what Google does is spam and abuse prevention. If you only indexed known-good sources I think the hard part would basically be done. Providing good search results, rich queries, sentiment analysis etc is all old hat by now.

"Known-good" sources have a serious habit of getting owned by malware and replaced by organized crime schemes. Anti-abuse can't be handled by whitelists, it needs to be an online system.

Garbage sources have a serious habit of getting taken over. It's not that common with quality sites, but also not impossible. A simple downvoting system would be good enough.

The problem is that even quality sites die. And when they do, their domain gets poached and suddenly it's not a quality site anymore. But according to the ranking algorithms it's the same site, since its quality is measured by how many links it gets.

Coopting quality sites is a bit of a cottage industry. There is also a significant amount of hacked wordpress sites. Their domain may be quite reputable, but they're unknowingly host to a ton of spam.

Old algorithm with todays internet content would be utterly garbage.

People keep saying that and yet the one person project search.marginalia.nu keeps delighting us.

I can't imagine they use Googles newest algorithm.

I'm actually using Google's original algorithm, with a modification suggested in the original PageRank paper called Personalized PageRank that allows using a specific segment of the internet as a tastemaker in creating the ranking.

It's a bit funny because the authors of the paper suggest this modification and explicitly states it makes the algorithm extremely resilient to manipulation from commercial interests.

what you want is the internet with only a couple hundred million people connected, only a million or so content creators.

Unless you have some pretty evil supervillain scheme, no moral compass, and succeed.... that world is never coming back.

I think that if the company doing the search results wasn't also the company getting money from SEO spammers posting ads on their websites then the search results might be better.

That old algorithm would just hand you a bunch of SEO spam.

I am not sure. It may be me lacking in understanding.

Google already does that though...

Yes but the Google algorithm circa 2003 would be much, much worse. There has been an arms race for the last 20 years.

The spooks will take over that one, too.

I've pondered this, too. Why not a DMOZ 2.0 type concept, where you seed the index with high quality sites and grow from there. Freshness won't matter as much for most topics, so crawling X times per day probably won't be necessary. Maybe have user defined flags to indicate the types of results they want (large sites, small sites, high authority, technical sites-- maybe different sort by features) with backlinks/domain prominence as a small factor, but also using NLP to determine authoritativeness of said content so as to facilitate less-linked site discovery.

That exists there: * https://curlie.org/

I think something that needs to be actively maintained like DMOZ is bound to burn out. I've been thinking about how you could passively maintain an index of good resources.

Imagine a service that provides you with a personal search engine in exchange for a list of your bookmarks. Those bookmarks provide the signal for what sites to index for the public search engine.

Neeva is worth looking at. I recently switched to it exclusively for a week and found it as good as Google for 90% of what I do. (FWIW, previous attempts doing this with Bing or DuckDuckGo last about a day before I get so mad I switch back.)

Right now Neeva seems very good at navigational queries, which I do 90% of the time. It's still not as good at deep research queries for obscure things. Probably related: Neeva is relying on Bing for a big chunk of their queries. But they are building their own index.

Neeva has the right business model (search as a product) which helps align interests. On the other hand some of their reasoning around the business execution left much to be desired:

https://www.reddit.com/r/vivaldibrowser/comments/pol41p/comm...

Just tried this, even signed up. Immediately they started force feeding me "news". Looks like they added me to some (daily!) mailing list. It looks like more of the same, but this time you pay for it.

Yeah, I tried Neeva for a few days, but it was slowing me down.

> Everytime one of these threads show up, I find myself wondering why someone doesn't launch a technical and research focused competitor. There seems to be such broad agreement that Google is now terrible for research that there's for sure a market for it.

Your bubble seems to agree, but the lack of serious competition, even in niches, is a sign that outside the HN bubble Google is in fact not seen as any worse.

Seems like for a decade several times a week this question shows up on HN and 99% of comments agree. Personally for my uses I don't remember it being significantly better, I'm not sure what I'm missing.

Not really. It's a sign that you can't profitably compete with Google.

DevonAgent may do what you want, it felt very thorough and powerful but I never got in the habit of using it, there’s a learning curve to it

https://www.devontechnologies.com/apps/devonagent#editions

mac os only. 5$ or 50$ for automation+archival

This site can help: http://teclis.com

I just searched 'what is energy' and is see posts from Quora and Medium on the first page. Definitely not what someone doing real research would be looking for

Depends on how you define "real research". Would someone doing real research ask that question?

It's analogous to predatory pricing. Google has all the necessary tech to go back to doing good searches. They can easily pivot back in a few months.

A new startup would need to dump loads of money into servers and building their own tech.

So there's just no way to recoup the cost of building a competitor.

You might be surprised how much compute it takes.

Not the same thing but, entire SO website runs on like 2 door sized racks. Search engine might be a different thing but if you have funding to get started, hardware costs aren’t going to be impossibly huge. Most is labor (engineering).

I’m curious, how much compute power it takes to index the whole web? I presume queries are super fast.

The monthly snapshots of CommonCrawl contains around 64k files of HTML, metadata and text only. The text only and metadata are around 250MB each, so in total the corpus would be around 16TB of text and 16TB of metadata. I don't know how much it costs to them, but when I used my own crawler I was able to download 500k webpages in around 4 hours (using the $80/month basic DigitalOcean server... let's say a total of $1/1M web pages, just the crawling process).

IMHO, indexing the whole web is an antifeature for technical and research topics. But there are projects like CommonCrawl that crawl a good portion of the web and give it away free.

what's the business model? Those verticals are probably not great for search ads. Do you think there is a large enough population of people willing to pay $X per month for a better technical + research search engine?