News for hackers. Stuff that matters.
Re:I tested four NVMe SSDs from four vendors – half lose FLU... by benlwalker on Monday February 21, 2022 @08:30PM (#30423043)
A flush command only guarantees, upon completion, that all writes COMPLETED prior to submission of the flush are non-volatile. Not all previously sent writes. NVMe base specification 2.0b section 7.1.

That's a very important distinction. You can't assume just because a write completed before the flush that it's actually durable. Only if it completed before you sent the flush.

I'm not very confident that software is actually getting this right all that often, although it probably is in this fsync test.

Re:I tested four NVMe SSDs from four vendors – half lose FLU... by wmf on Monday February 21, 2022 @10:05PM (#30423596)
Is there a separate barrier command so you don't have to track all the writes individually in software?
Re:I tested four NVMe SSDs from four vendors – half lose FLU... by benlwalker on Monday February 21, 2022 @10:20PM (#30423661)
Nope. The reason this is so complex is that these devices are actually highly parallel machines with multiple queues accepting commands. It's quite difficult to even define "before" in terms of command sequence. For example, if you have a device with two hardware queues for submitting commands and a software thread for each, if you submit a flush on one queue, which commands on the other queue does it affect?

Or what if the device issues a pci write to the completion entry that passes a flush command being submitted on the wire?

I think the only interpretation that makes sense is from the perspective of a single software thread. If that particular thread has seen the completion via any mechanism and then that thread issues the flush, then you know the write is durable. Other than that, the device makes no promises.