Re:Unified Graph-Vector Database for AI Retrieval
by xavcochran on Friday May 30, 2025 @12:47PM
(#44137956)
In this post I want to talk about my experience building a database using RocksDB and why I switched to LMDB.
When I first started building HelixDB (https://helix-db.com), a graph-vector database written in Rust, I didn’t know which storage engine to use. After some chatgpt-driven development (lol) I went with RocksDB.
RocksDB worked well and we were able to build it quickly. Initial benchmarks showed good speed for both reads and writes; all was looking fine. Then came the time to deploy Helix in the cloud for a customer. During development, I was building on my mb pro so build times were not excruciating (~7 mins). However, in the cloud, we were deploying Helix on small aws instances. It would take over 40 minutes for Helix to build in the cloud, and sometimes it would just crash.
I started looking for a fix. Given graph DBs are read heavy when it comes to traversals, all things pointed to LMDB being a good option. I spent 6 hours changing over the graph engine to use LMDB under the hood. On the first build it took 40 seconds on my macbook, a 90% decrease, and in the cloud it was now building in ~4 mins. We also got a 60% read performance increase due to LMDB’s B-Tree storage architecture. Now I know RocksDB is still the king for writes and can be written to in parallel whereas LMDB can only be written to synchronously. At first the write slowdown was a worry, however, I was able to make some key design decisions that meant we are still able to write millions of nodes and edges to helix per second, save a huge amount of space on disk, and make reads even faster for graph and vector traversals.
The first thing I did was change the way we were storing keys. Originally we were naïvely storing keys as UUID strings which take up 36 bytes each. I changed this to UUIDv6 unsigned 128 bit integers which take up 16 bytes each (56% storage space saving). This change also allowed for something else. UUID v6 keys include the time of creation as part of the key meaning keys are lexicographically ordered based on creation time. This allowed me to use LMDB’s put flags (APPEND for nodes and edges and APPEND_DUP for direction storage) which makes LMDB insert the new item at the end of the db rather than traversing the tree internally to find the right place for it (APPEND_DUP also stores items with duplicate keys under one key instead of many).
This led to an extreme speed up of both reads and writes. Our current write speed is up to 5 million nodes per second and up to 700k edges per second. Note that the difference between the nodes and the edges is due to the fact we are storing the direction of each edge for each node separately. While this slows down the edge insertion (without inserting the direction, edge insertion is 5 million per second), it massively increases the read performance for edge traversals, which is a payoff we think is necessary. Our 1-hop performance is great as well with a performance with a performance of 5-100 μs (microseconds) per hop (dependant on start node) on a dataset of 65 million nodes and 1.9 billion edges. Note that all bulk reads are currently single threaded in HelixDB and we’re currently implementing parallel LMDB iterators via the Heed library with the guys from Meilisearch.
So with all that being said, I think LMDB wins the storage engine battle with the caveat that it is completely down to preference, workload and implementation.
Quickly touching on the vector side of Helix, from these performance improvements, our insertion performance is at around 2-5ms (dependent on the vector) and 3ms per vector search (with cosine similarity with 1M vectors).
Would love to hear more about people’s experience with LMDB, RocksDB or another storage engine, especially if people have found crazy performance boosts/hacks.
When I first started building HelixDB (https://helix-db.com), a graph-vector database written in Rust, I didn’t know which storage engine to use. After some chatgpt-driven development (lol) I went with RocksDB.
RocksDB worked well and we were able to build it quickly. Initial benchmarks showed good speed for both reads and writes; all was looking fine. Then came the time to deploy Helix in the cloud for a customer. During development, I was building on my mb pro so build times were not excruciating (~7 mins). However, in the cloud, we were deploying Helix on small aws instances. It would take over 40 minutes for Helix to build in the cloud, and sometimes it would just crash.
I started looking for a fix. Given graph DBs are read heavy when it comes to traversals, all things pointed to LMDB being a good option. I spent 6 hours changing over the graph engine to use LMDB under the hood. On the first build it took 40 seconds on my macbook, a 90% decrease, and in the cloud it was now building in ~4 mins. We also got a 60% read performance increase due to LMDB’s B-Tree storage architecture. Now I know RocksDB is still the king for writes and can be written to in parallel whereas LMDB can only be written to synchronously. At first the write slowdown was a worry, however, I was able to make some key design decisions that meant we are still able to write millions of nodes and edges to helix per second, save a huge amount of space on disk, and make reads even faster for graph and vector traversals.
The first thing I did was change the way we were storing keys. Originally we were naïvely storing keys as UUID strings which take up 36 bytes each. I changed this to UUIDv6 unsigned 128 bit integers which take up 16 bytes each (56% storage space saving). This change also allowed for something else. UUID v6 keys include the time of creation as part of the key meaning keys are lexicographically ordered based on creation time. This allowed me to use LMDB’s put flags (APPEND for nodes and edges and APPEND_DUP for direction storage) which makes LMDB insert the new item at the end of the db rather than traversing the tree internally to find the right place for it (APPEND_DUP also stores items with duplicate keys under one key instead of many).
This led to an extreme speed up of both reads and writes. Our current write speed is up to 5 million nodes per second and up to 700k edges per second. Note that the difference between the nodes and the edges is due to the fact we are storing the direction of each edge for each node separately. While this slows down the edge insertion (without inserting the direction, edge insertion is 5 million per second), it massively increases the read performance for edge traversals, which is a payoff we think is necessary. Our 1-hop performance is great as well with a performance with a performance of 5-100 μs (microseconds) per hop (dependant on start node) on a dataset of 65 million nodes and 1.9 billion edges. Note that all bulk reads are currently single threaded in HelixDB and we’re currently implementing parallel LMDB iterators via the Heed library with the guys from Meilisearch.
So with all that being said, I think LMDB wins the storage engine battle with the caveat that it is completely down to preference, workload and implementation.
Quickly touching on the vector side of Helix, from these performance improvements, our insertion performance is at around 2-5ms (dependent on the vector) and 3ms per vector search (with cosine similarity with 1M vectors).
Would love to hear more about people’s experience with LMDB, RocksDB or another storage engine, especially if people have found crazy performance boosts/hacks.
PS: We love feedback :)
See our code and our optimizations here: https://github.com/HelixDB/helix-db