Show HN: A discovery-focused search engine for Hacker News
We (Nick, Dens, Denzell, Fede, Drew, Aaryan, and Daniel) have been building HN Discovery, a discovery-focused search engine for Hacker News, in our spare time for the past 6 months and are excited to show it! It adds the following features relative to the existing keyword search interface and preserves the existing ones:
- no-JS version (hnnojs.trieve.ai)
- site:{required_site} and site:{negated-site} filters
- public analytics
- LLM generated query suggestions based on random stories
- recommendations
- dense vector semantic search
- SPLADE fulltext search
- RAG AI chat
- order by descendant count
client code (FOSS self-hostable) - https://github.com/devflowinc/trieve-hn-discovery
engine code (BSL source-available) - https://github.com/devflowinc/trieve
There is an extended about page with detailed information on features, how much it costs to run, etc. here - https://hn.trieve.ai/about.
(
34 comments
| 60 points
| on HN
)
I searched for `Excel` since that's a topic I care about and tend to follow frequently.
The first ("most relevant") link is a post with 1 point and 0 comments from 2020. The second link has 1 comment and 2 points, from 2017.
The top two links on Algolia's search have about 1000 points each and are way more topical.
I tried to hit "Back" to make this comment, and saw the site broke my browser's navigation. I was forced to right click on "Back" (or spam the back button) to get back here... so not a great experience, overall
- Algolia ranks by points by default while we are ranking by relevance score which is the difference. You can order our results by points if you want to with the order by select component that says "relevance".
We went back and forth on making points sorting default and ended up deciding against it, but maybe we should have. Our thinking was that since it's focused on "discovery" it was worth prioritizing relevance, but I can see how it can feel the result quality isn't as great. HN is really good at highlighting interesting links.
Best fix would have been LTR, but we made incorrect decisions early on which made the rewrite a bit too hard - https://trieve.ai/launching-trieve-hn-discovery/#relevance-q...
I've bookmarked this site and hope it remains available when I need it, unlike many great Show HN posts that vanish after six months or so.
Though short comments seemed to score a bit too highly IMHO. It took a while to find a query that found the long rambly comment I needed.
This seems way higher than I expected. Cloud pricing is out of control when Postgres is already > $500 for a small instance that could be run for a fraction if it wouldn’t be a cloud provider.
That is, if this 6k number is actually true. Part of me (forgive me) is in fact wondering if maybe this is an advertisement for your SaaS and you’re inflating this number to make people think there’s no way they can build a thing like that themselves. But, giving you the benefit of doubt, if you are truly paying this, you’re overspending by more than an order of magnitude. Most likely too many middlemen.
Email is in my profile if you want to talk about it.
Our terraform and helm are public in the repo - https://github.com/devflowinc/trieve/tree/main/terraform/gcl...
Speeding up ingest made experimenting with different indexing strategies a lot more viable.
But when I search for "FreebBSD" I get:
* FreeBSD is an amazing operating system
* FreeBSD Is an Operating System
* 9x FreeBSD – a lesson in poor defaults
In the top 15 results, 10 are duplicates. And none of the articles are interesting.
If I sort by points instead of relevance, nothing has to do with FreeBSD.
We still are not deduping because I think that's semi-useful for "past" behavior, but I'm happy that the sort functionality is no longer broken.
Some questions (and thanks for the detailed "about" page, it answered several of my initial questions!) ~
Will you re-index and keep updating the system to improve the quality of results, or what is the plan? It'd be awesome to have something more nuanced than Algolia which stays updated in near real time, like Algolia.
How easy or challenging is it to bootstrap / re-index? Is it possible to ingest new data with partial updates to the existing indices, or is a full indexing from zero always required?
Are GPUs strictly necessary? Is it possible to use only CPUs if indexing speed isn't a great concern?
Does it really require a terabyte of working memory to index and serve all of the data for HN? (4x 256GB / 128 CPUs is mentioned in your ops details) This is a lot of resources! Like A LOT!
Have you thought considered indexing other high-quality data sources? For example:
* Lobste.rs (I think you can email them requesting a DB dump, they want disclosures about the intended purpose)
* Slashdot (debatable quality, but goes back 27 years which could be interesting)
* Review sites: Chipsandcheese, TomsHardware, Anandtech, HardOCP
* Lwn, Phoronix
* I'm surely missing other good ones, the discussion in The End of Anandtech article from today mentions a bunch of interesting sources: https://news.ycombinator.com/item?id=41399872
I wonder if getting some of these data sources through CommonCrawl or archive.org would reduce the crawl+parse annoyance?
At some point I want to put together an HN-Awesome-Search page which covers all the custom search indexes HN folks have made over the years.
Thank you!
THANK YOU! Initial motivation for us was, to a large extent, making it easier to throw data into an API and test to see if all the latest AI buzzword tech was useful for your problem. Hearing that we've done that in a semi-usable way is energizing!
> re-index and keep updating
Yes. It updates in real-time and we are collecting CTR data and analytics to continually fine-tune. There's also a "Rate Results" button which we are looking at the feedback from to improve.
> partial updates
Partial updates work really well and we are doing it constantly with no impact to performance or uptime. Full indexing from zero is only required if we want to make an index-wide change to improve relevance.
> GPUs strictly necessary? Is it possible to use only CPUs if indexing speed isn't a great concern?
They are not strictly required. But, inference on CPU is going to be 300+ms at minimum which makes the search feel very laggy. Indexing on CPU is actually less painful than search, because it's lot more ok for that to be slow.
> Does it really require a terabyte of working memory to index and serve all of the data for HN? (4x 256GB / 128 CPUs is mentioned in your ops details)
We didn't build our own db engine solution and are running Qdrant for this. We certainly tried to get memory usage down and this is the best we got. Qdrant's calculator says this should only take 128gb's with Dense vectors and binary quantization, but we just didn't find that to be true. However, it may be the case that we can improve this further with time.
> indexing other high-quality data sources
Maybe? I'm somewhat active on lobste.rs, but don't partipate much in the other communities which reduces my interest. HN is really cool since I understand it enough to get a feel for how good the search is which helps us tune things for the overall product.
> HN-Awesome_Search page
You should! Checkout out the last section of our "History of HN Search" blog to get a starting point at least --> https://trieve.ai/history-of-hnsearch/
Tried to be very thorough.
Has some additional features for browsing and can find similar users also.