Pebble: A RocksDB Inspired Key-Value Store Written in Go

Re:Pebble: A RocksDB Inspired Key-Value Store Written in Go by itsbilal on Tuesday September 15, 2020 @04:23PM (#24486298)

Hi, I'm on the team that works on Pebble. Partially synced WAL records are easier to detect, as they would just appear as corrupt records and we can stop WAL replay at that point. Non-WAL writes are even easier to handle as SSTable files are immutable once fully written and synced. We rely pretty heavily on fsync/fdatasync calls to guarantee that "all" the data in a given range made it.

In addition to randomized crash tests, we have a suite of end-to-end integration tests on top of Cockroach, called Roachtests, that put clusters under a combination of node crash/restart scenarios and confirm data consistency.

[ Parent | on HN ]

Re:Pebble: A RocksDB Inspired Key-Value Store Written in Go by ignoramous on Tuesday September 15, 2020 @05:05PM (#24486691)

> We rely pretty heavily on fsync/fdatasync calls to guarantee that "all" the data in a given range made it.

Reminds me of fsync gate: https://news.ycombinator.com/item?id=20491965 and https://news.ycombinator.com/item?id=19119991 (not implying Pebble uses fysnc incorrectly).

[ Parent | on HN ]

Re:Pebble: A RocksDB Inspired Key-Value Store Written in Go by diehunde on Wednesday September 16, 2020 @06:23PM (#24498706)

Hi! Awesome work.

One of the cool things about the LevelDB codebase is the `Repository contents` section. A brief description of the most relevant modules so people can get familiarized with the code base quicker. As someone very interested in storage engines, I would love to see something similar here. Are you guys planning on adding some extra documentation to the project?

[ Parent | on HN ]