News for hackers. Stuff that matters.
Re:Show HN: Search 1M Linux kernel commit messages by java-man on Thursday May 13, 2021 @10:36AM (#27142537)
Quite fast.

Is there a document/whitepaper which describes how it works?

Re:Show HN: Search 1M Linux kernel commit messages by karterk on Thursday May 13, 2021 @10:53AM (#27142742)
We don't have a proper design document (but I certainly think we should have one). I will try to offer a high level summary:

At the heart of Typesense is a `token => documents` inverted index backed by an Adapative Radix Tree (https://db.in.tum.de/~leis/papers/ART.pdf), which is a memory-efficient implementation of the Trie data structure. ART allows us to do fast fuzzy searches on a query.

All indices are stored in-memory, while the documents are stored on disk on RocksDB. All underlying data structures were carefully designed, benchmarked and optimized to exploit cache locality and utilize all cores efficiently.

Re:Show HN: Search 1M Linux kernel commit messages by hpeinar on Thursday May 13, 2021 @02:47PM (#27145751)
Is there an estimate how much memory does this site need to have the full index in memory?

I gotta say, I've seen at least one? other Typesense post here on HN at some point and I can't really comprehend HOW FAST this actually is, especially considering how much more bloat and slower general web has gone in the past years.

I don't really have anything to search for from the given site but I just played around with it to enjoy the speed.

Re:Show HN: Search 1M Linux kernel commit messages by jabo on Thursday May 13, 2021 @03:43PM (#27146424)
Ha! Once you're used to instant search-as-you-type experiences, it's pretty addictive. From a UX perspective, it actually helps increase engagement as people tend to search for more queries, since there's no cognitive overhead in typing something and waiting for a result.

The commits data is ~950MB on disk, with ~1 million records. It takes up about ~3GB in RAM when indexed in Typesense.