Are You Sure You Want to Use MMAP in Your Database Management System?

Re:Are You Sure You Want to Use MMAP in Your Database Manage... by pclmulqdq on Sunday July 02, 2023 @10:02PM (#36567830)

I worked with the Bigtable folks at Google. LevelDB's design is ripped straight from BigTable, which was designed with that assumption in mind. I'm also pretty sure it was not designed specifically for Google Chrome's use case - it was written to be a general key-value storage engine based on BigTable, and Google Chrome was the first customer.

RocksDB is Facebook's offshoot of LevelDB, basically keeping the core architecture of the storage engine (but multithreading it), and is used internally at Facebook as the backing store for many of their database systems. I have never heard from anyone that RocksDB was optimized for in-memory workloads at all, and I think most benchmarks can conclusively say the opposite: both of those DB engines are pretty bad for workloads that fit in memory.

[ Parent | on HN ]

Re:Are You Sure You Want to Use MMAP in Your Database Manage... by hyc_symas on Sunday July 02, 2023 @10:18PM (#36567922)

I think we've gone off on a tangent. At any rate, both LevelDB and RocksDB are still single-writer so whatever point seems to have been lost along the way.

[ Parent | on HN ]

Re:Are You Sure You Want to Use MMAP in Your Database Manage... by earthnail on Monday July 03, 2023 @04:05AM (#36570134)

I've used RocksDB for an in-memory K/V store of ~600GB in size and it worked really well. Not saying it's the best choice out there but it did the job very well for us. And in particular because our dataset was always growing and we needed the option to fallback to disk if needed, RocksDB worked very well.

Was a PITA to optimise though; tons of options and little insight into which ones work.

[ Parent | on HN ]