News for hackers. Stuff that matters.
Re:Apple's custom NVMes are amazingly fast – if you don't ca... by monocasa on Thursday February 17, 2022 @11:05AM (#30374585)
Last time I checked (which is a while at this point, pre SSD) nearly all consumer drives and even most enterprise drives would lie in response to commands to flush the drive cache. Working on a storage appliance at the time, the specifics of a major drive manufacturer's secret SCSI vendor page knock to actually flush their cache was one of the things on their deepest NDAs. Apparently ignoring cache flushing was so ubiquitous that any drive manufacturer looking to have correct semantics would take a beating in benchmarks and lose marketshare. : \

So, as of about 2014, any difference here not being backed by per manufacturer secret knocks or NDAed, one-off drive firmware was just a magic show, with perhaps Linux at least being able to say "hey, at least the kernel tried and it's not our fault". The cynic in me thinks that the BSDs continuing to define fsync() as only hitting the drive cache is to keep a semantically clean pathway for "actually flush" for storage appliance vendors to stick on the side of their kernels that they can't upstream because of the NDAs. A sort of dotted line around missing functionality that is obvious 'if you know to look for it'.

It wouldn't surprise me at all if Apple's NVME controller is the only drive you can easily put your hands on that actually does the correct things on flush, since they're pretty much the only ones without the perverse market pressure to intentionally not implement it correctly.

Since this is getting updoots: Sort of in defense of the drive manufacturers (or at least stating one of the defenses I heard), they try to spec out the capacitance on the drive so that when the controller gets a power loss NMI, they generally have enough time to flush then. That always seemed like a stretch for spinning rust (the drive motor itself was quite a chonker in the watt/ms range being talked about particularly considering seeks are in the 100ms range to start with, but also they have pretty big electrolytic caps on spinning rust so maybe they can go longer?), but this might be less of a white lie for SSDs. If they can stay up for 200ms after power loss, I can maybe see them being able to flush cache. Gods help those HMB drives though, I don't know how you'd guarantee access to the host memory used for cache on power loss without a full system approach to what power loss looks like.

Re:Apple's custom NVMes are amazingly fast – if you don't ca... by garaetjjte on Thursday February 17, 2022 @03:21PM (#30378138)
Flush with other vendors at least does something as they block for some time too, just not as long as Apple.

Apple implementation is weird because actual amount of data written doesn't seem to affect flush time.

Re:Apple's custom NVMes are amazingly fast – if you don't ca... by monocasa on Thursday February 17, 2022 @04:17PM (#30378811)
On at least one drive I saw, the flush command was instead interpreted as a barrier to commands being committed to the log in controller DRAM, which could cut into parallelization, and therefore throughput, looking like a latency spike but not a flush out of the cache.
Re:Apple's custom NVMes are amazingly fast – if you don't ca... by marcan_42 on Thursday February 17, 2022 @11:12PM (#30382060)
My test is single threaded, and thus has no parallelism to begin with.
Re:Apple's custom NVMes are amazingly fast – if you don't ca... by monocasa on Thursday February 17, 2022 @11:16PM (#30382078)
The drive controller is internally parallel. The write is just a job queue submission, so the next write hits while it's still processing previous requests.
Re:Apple's custom NVMes are amazingly fast – if you don't ca... by marcan_42 on Friday February 18, 2022 @12:30AM (#30382449)
People have tested this stuff on storage devices with torture tests. Can you point at an example of a modern (directly attached) NVMe drive from a reputable vendor that cheats at this?

FWIW, macOS also has F_BARRIERFSYNC, which is still much slower than full syncs on the competition.

Re:Apple's custom NVMes are amazingly fast – if you don't ca... by SamReidHughes on Thursday February 17, 2022 @10:51PM (#30381953)
On my benchmarking of some consumer HDD's, back in 2013 or so, the flush time was always what you'd expect based on the drive's RPM. I got no evidence the drive was lying to me. These were all 2.5" drives.

My understanding was, the capacitor thing on HDD's is to ensure it completely writes out a whole sector, so it passes the checksum. I only heard the flush cache thing with respect to enterprise SSD's. But I haven't been staying on top of things.

Re:Apple's custom NVMes are amazingly fast – if you don't ca... by monocasa on Thursday February 17, 2022 @11:13PM (#30382063)
You can't base spinning rust flushes on RPM; the seek arm is what dominates.
Re:Apple's custom NVMes are amazingly fast – if you don't ca... by SamReidHughes on Thursday February 17, 2022 @11:23PM (#30382123)
Not at all, and it certainly wasn't relevant to the benchmarking I was doing, as I was focusing on writes to and fsync performance for the same track.
Re:Apple's custom NVMes are amazingly fast – if you don't ca... by monocasa on Thursday February 17, 2022 @11:36PM (#30382178)
You definitely weren't testing the cache in a meaningful way if you were hovering over the same track.

WRT to the capacitor thing being about a single sector, think about the time spans. You should be able to even cut out the drive motor power, and still stay up for 100s of ms. In that time you can seek to a config track and blit out the whole cache. If you're already writing a sector you'll be done in microseconds. The whole track spins around every ~8ms at 7200RPM.

Re:Apple's custom NVMes are amazingly fast – if you don't ca... by exikyut on Friday February 18, 2022 @12:49AM (#30382522)
Tangential thinking out loud: this makes me think of a sort of interleaving or striping mechanism that tries to leave a small proportion of every track empty, such that ideal power loss flush scenarios would involve simply waiting for the disk to spin around to the empty/reserved area in the current track. On drives that aren't completely full, it's probably statistically reasonable that for any given track position there's going to be a track with some reserved space very close by, such that the amount of movement/power needed to seek there is smaller.

Of course, this approach describes a completely inverted complexity scenario in terms of sector remapping management, with the size of the associated tables probably being orders of magnitude larger. :<

Now I wonder how much power is needed for flash writes. The chances are an optimal-and-viable strategy would probably involve a bit of multichannel flash on the controller (and some FEC because why not).

Oooh... I just realized things'll get interesting if the non-volatile RAM thing moves beyond the vaporware stage before HDDs become irrelevant. Last-millimeter write caching will basically cease to be a concern.

But thinking about the problem slightly more laterally, I don't understand why nobody's made inline SATA adapters with RAM, batteries and some flash in them. If they intercept all writes they can remember what blocks made it to the disk, then flush anything in the flash at next power on. Surely this could be made both solidly/efficiently and cheaply...?

Re:Apple's custom NVMes are amazingly fast – if you don't ca... by culpable_pickle on Friday February 18, 2022 @02:56AM (#30383211)
> But thinking about the problem slightly more laterally, I don't understand why nobody's made inline SATA adapters with RAM, batteries and some flash in them.

Hardware raid controllers with Battery Backup Units was really popular starting in the mid 90’s until maybe mid 2010’s? Software caught up in a lot of features and batteries failed often and required a lot more maintenance. Super caps were to replace the batteries but I think SSDs and software negated a ton of the value add. You can still buy them but they’re pretty rare to see in the wild.

Re:Apple's custom NVMes are amazingly fast – if you don't ca... by exikyut on Friday February 18, 2022 @10:40AM (#30387122)
I've heard of those, they sound mildly interesting to play with, if just to go "huh" at and move on. I get the impression the main reason they developed a... strained reputation was their strong tendency to want to do RAID things (involving custom metadata and other proprietaryness) even for single disks, making data recovery scenarios that much more complicated and stressful if it hadn't been turned off. That's my naive projection though, I (obviously) have no real-world experience with these cards, I just knew to steer far away from them (heh)

An inline widget (SATA on both sides) that just implements a write cache and state machine ("push this data to these blocks on next power on") seems so much simpler. You could even have one for each disk and connect to a straightforward RAID/SAS controller. (Hmm, and if you externalize the battery component, you could have one battery connect to several units...)

You are indeed right about the battery/capacitor situation ("you have to open the case?!"), I wouldn't be surprised if the battery level reporting in those RAID cards was far from ideal too lol

With all this being said, a UPS is by far the simplest solution, naturally, but also the most transiently expensive.