Probably more like downvoted because missing the point.

Sure fsync allows that behavior, but also it's so widely misunderstood that a lot of programs which should do a "full" flush only do a fsync, including Benchmarks. In which case they are not comparable and doing so is cheating.

But that's not the point!

The point is that with the M1 Macs SSDs the performance with fully flushing to disk is abysmal bad.

And as such any application with cares for data integrity and does a full flush can expect noticable performance degradation.

The fact that Apple neither forces frequent full syncs or at least full syncs when a Application is closed doesn't make it better.

Though it is also not surprising as it's not the first time Apple set things up under the assumption their hardware is unfailable.

And maybe for a desktop focused high end designs where most devices sold are battery powered that is a reasonable design choice.

"And maybe for a desktop focused high end designs where most devices sold are battery powered that is a reasonable design choice"

Does the battery last forever? Do they never shut down from overheating, shut down from being too cold, freeze up, they are water and coffee proof?

Talk to anyone that repairs mac about how high-end and reliable their designs trully are - they are better than bottomn of the barrel craptops, sure, but not particularly amazing and have some astounding design flaws.

As the article points out, a lot of those cases can be detected with advanced notice (dying battery, and overheating - probably even being too cold). In those cases the OS makes sure all the caches are flushed.

Spilled drinks are a viable cause for concern, but if they do enough damage to cause an unexpected shutdown, you've probably got bigger issues than unflushed cache.

"In those cases the OS makes sure all the caches are flushed."

I have never heard of this functionality being included in any major OS, could you please provide some reference to this being documented?

I think that's misleading.

On many laptops even with water damage you can recover your local data fully, not do for Macs (for more reasons then just data loss/corruption due to non flushing).

Especially if you are already in a bad situation you don't want your OS to make it worse.

How cold is too cold for a computer?

The CPU can't possibly get too cold. See for example overclocking performed by cooling the CPU with liquid nitrogen. Condensation is a factor as is lost of ductility of plastic at low temp making it brittle. Expansion and contraction of materials especially when different materials expand to different degrees.

"The CPU can't possibly get too cold" - Untrue. There are plenty of chips with what overclockers like to call "cold bugs".

Sequential logic (flipflops) has a setup time requirement. This means the combinatorial computation between any two connected pairs of flops (output of flop A to input of flop B) has to do its job fast enough such that the input of B stops toggling some amount of time before the next clock edge arrives at the flipflop. Violate that timing, and B will sometimes sample the wrong value, leading to an error.

Setup time is what most people are thinking about when they use LN2 or other exotic forms of cooling. By cooling things down, you usually improve the performance of combinatorial logic, which provides more setup time margin, allowing you to increase clock speed until setup time margin is small again.

But flops also have hold time requirements - their inputs have to remain stable for some amount of time after the clock edge, not just before. It's here where we can run into problems if the circuit is too cold. Imagine a path with relatively little combinatorial logic, and not much wire delay. If you make that path too fast, it might start violating hold time on the destination flop. Boom, shit doesn't work.

Many phones, laptops cameras and similar are only guaranteeing functionally by above 0 degree....

Luckily they often operate in lower temperatures too, but not seldomly by hoping they don't get cooled that much themself (because they are e.g. in your pocket).

0 degree what?

Fahrenheit, Celsius, horizontal angle?

The biggest thing is the battery. The CPU doesnt get too cold, but batteries degrade or stop performing when they get too cold.

Edit: For actual temperatures, in my experience its when the device is in use for a sustained amount of time in under 10f weather

Incidentally CPUs do get too cold, not at a reasonable temperature, but sufficiently low temperatures do change the characteristics of semi conductors. Not something to worry about if you're not using liquid nitrogen (or colder).

I've had my phone shut off on me from being out in the Chicago cold for a couple hours. Battery over 50% when I brought it back inside and warmed it up.

If i go ousode in winter, the bsttery dies around zero degrees. Keep in mind that you laptop could be in a bag in sleep mode or idle

I mean the Apple hardware in question is usually a laptop, which has its own very well instrumented battery backup. In most cases the hardware knows well in advance if the battery is gonna run dry.

And yes the hardware is failable. But the kind if failure that would cause the device to completely lose power is extremely rare. The OS has many chances to take the hint and flush the cache before powering down.

Note: this is pure conjecture.

> The point is that with the M1 Macs SSDs the performance with fully flushing to disk is abysmal bad.

How sure are we the drives that flush caches more quickly are actually flushing the caches?

Good Point.

A simple test can be to see the degree of dataloss you can occur with a hard power off.

I think the author did that test for M1 Mac but idk. if they did the test with the other laptops.

But then the M1 Mac is slower when flushing then most SSDs out there and even some HDDs. I think if most SSDs wouldn't flush data at all we would know of that and I should have run into problems with the few docent hard resets I ran into in the last few years. (And sure there are probably some SSDs which cheap out on cache flushing in a dangerous way, but most shouldn't as far as I can tell).

We’d see data loss only if the power loss or hard reset happened before the data is actually flushed. After the data is accepted into the buffer there would be a narrow time window when it could occur. Also, a hard reset on the computer side may not be reflected on the storage embedded electronics.

What is worse is their NVMe controller having 50x worse flush performance than the competition.

The competitions controller may be ignoring the F_FULLFSYNC. This is a known issue which is why apple have approved vendors for mac pro drives.

It isn't, because otherwise it would be showing the ~same performance with and without sync commands, as I showed in the thread. There is a significant performance loss for every drive, but Apple's is way worse.

There is no real excuse for a single sector write to take ~20ms to flush to NAND, all the while the NAND controller is generating some 10MB/s of DRAM traffic. This is a dumb firmware design issue.

It may be interpreting it differently. You arent comparing apples to apples, quite literally.

Why not compare macOS and linux on approved x86 mac hardware. i.e. fusion drive or whatever.

Also, as suggested - try F_BARRIERFSYNC, which flushes anything before the barrier (used for WAL IIRC).

This affects T2 Macs too, which use the same NVMe controller design as M1 Macs.

We've looked at NVMe command traces from running macOS under a transparent hypervisor. We've issued NVMe commands outside of Linux from a bare-metal environment. The 20ms flush penalty is there for Apple's NVMe implementation. It's not some OS thing. And other drives don't have it. And I checked and Apple's NVMe controller is doing 10MB/s of DRAM memory traffic when issued flushes, for some reason (yes, we can get those stats). And we know macOS does not properly flush with just fsync() because it actively loses data on hard shutdowns. We've been fighting this issue for a while now, it's just that it only just hit us yesterday/today that there is no magic in macOS - it just doesn't flush, and doesn't guarantee data persistence, on fsync().

Ive just been scanning through linux kernel code (inc ext4). Are you sure that its not issuing a PREFLUSH? What are your barrier options on the mount? I think you will find these are going to be more like F_BARRIERFSYNC.

I couldnt find much info about it - but the official docs are here: https://kernel.org/doc/html/v5.17-rc3/block/writeback_cache_...

Those are Linux concepts. What you're looking for is the actual NVMe commands. There's two things: FLUSH (which flushes the whole cache), and a WRITE with the FUA bit set (which basically turns that write into write-through, but does not guarantee anything about other commands). The latter isn't very useful for most cases, since you usually want at least barrier semantics if not a full flush for previously completed writes. And that leaves you with FLUSH. Which is the one that takes 20ms on these drives.

> Those are Linux concepts. What you're looking for is the actual NVMe commands.

Im not sure what commands are being sent to the NVMe drive. But what you are describing as a flush would be F_BARRIERFSYNC - NOT the F_FULLFSYNC which youve been benchmarking.

Sigh, no. A barrier is not a full flush. A barrier does not guarantee data persistence, it guarantees write ordering. A barrier will not make sure the data hits disk and is not lost on power failure. It just makes sure that subsequent data won't show up and not the prior data, on power failure. NVMe doesn't even have a concept of barriers in this sense. An OS-level barrier can be faster than a full sync only because it doesn't need to wait for the FLUSH to actually complete, it can just maintain a concept of ordering within the OS and make sure it is maintained with interleaved FLUSH calls.

I don't know why you keep pressing on this issue. macOS has the same performance with F_FULLFSYNC as Linux does with fsync(). Why would they be different things? We're getting the same numbers. This entire thing started because fsync() on these Macs on Linux was dog slow and we couldn't figure out why macOS was fast. Then we found F_FULLFSYNC which has the same semantics as fsync() on Linux. And now both OSes perform equally slowly on this hardware. They're obviously doing the same thing. And the same thing on Linux on non-Apple SSDs is faster. I'm sure I could install macOS on this x86 iMac again and show you how F_FULLFSYNC on macOS also gives better performance on this WD drive than on the M1, but honestly, I don't have the time for that, the isssue has been thoroughly proved already.

Actually, I have a better one that won't waste as much of my time.

Plugs in a shitty USB3 flash drive into the M1.

224 IOPS with F_FULLFSYNC. On a shitty flash drive. 58 IOPS with F_FULLFSYNC. On internal NVMe.

Both FAT32.

Are you convinced there's a problem yet?

(I'm pretty sure the USB flash drive has no write cache, so of course it is equally fast/slow with just fsync(), but my point still stands - committing writes to persistent storage is slower on this NVMe controller than on a random USB drive)

OK - thanks for humouring me marcan. Sorry to waste your time. Clearly something is not right here.

Thank you, you've made this very clear for me.

It seems to be pretty apples to apples, they're running the same benchmark using equivalent data storage APIs on both systems. What are you thinking might be different? The Linux+WD drive isn't making the data durable? Or that OSX does something stupid which could be the cause of the slowdown rather than the drive? Both seem implausible.

I downvoted you because you complained about your downvotes.

How did that work out for you?