The LMDB test is a bit contrived. By choosing a key of 16 bytes and a value of 4000 bytes, it's nearly optimal. The underlying block device wants 4KiB aligned writes for best performance, and that's pretty much what this test is doing. RocksDB is best at performing writes into very small records, where it can coalesce them and reduce write amplification. LMDB can't do this. As the size of the records increases, the RocksDB performance advantage disappears, and then it eventually loses out. In my own tests, I've found that the crossover point is about 250 bytes. A better test comparing LMDB to RocksDB would show what this crossover is, in a more controlled fashion than what I have done.

That's definitely a problem because the same yet to understand pattern may happen later with lmdb if there are the conditions (given that they are not understood) and create a random performance issue.

I have zero ideas how any of these work underneath, but out of a wild guess, is there any chance that alignment (or similar) is the issue? e.g. maybe RocksDB tries to align its writes to some boundary in a way that's awful for Optane's technology?

Alignment sounds like a plausible factor, but in my testing I've found that the penalty for unaligned writes on the Optane P4800X is less than a factor of two.

Hadn't really thought of that. I suppose we could have retried by making a single partition taking up the entire space, and letting fdisk take care of aligning the start of the partition. But I assumed that alignment wouldn't be an issue when using the unpartitioned block device.