News for hackers. Stuff that matters.
Re:Apple's custom NVMes are amazingly fast – if you don't ca... by steeve on Thursday February 17, 2022 @03:16AM (#30370691)
As the article mentions, on laptops, this is pretty clever. On desktops though...

Perhaps real macs should be equipped with internal batteries to flush to disk in the case of power loss?

I think I heard some enterprise motherboards/controllers/computers did just that, given the upside in normal operation.

Re:Apple's custom NVMes are amazingly fast – if you don't ca... by marcan_42 on Thursday February 17, 2022 @03:24AM (#30370738)
These machines are actually low-power enough that you could implement a last-gasp flush mechanism. The Mac Mini already survives 1-2 seconds without AC power (at least if idle). You could plausibly detect AC power being yanked and immediately power down all downstream USB/TB3 devices and the display (on iMacs), freeze all CPUs into idle, and have plenty enough reservoir cap to let NVMe issue a flush.

But they aren't doing that. I tested it on the Mac Mini. It loses several seconds of fsync()ed data on hard shutdown.

This does require a last-gasp indication from the PSU to the rest of the system, so if they don't have that, it's not something they could add in a firmware update.

Re:Apple's custom NVMes are amazingly fast – if you don't ca... by rubatuga on Thursday February 17, 2022 @04:22AM (#30371131)
I mean the ATX standard has this signal built in, so Apple could just copy it:

https://en.wikipedia.org/wiki/Power_good_signal

Re:Apple's custom NVMes are amazingly fast – if you don't ca... by dale_glass on Thursday February 17, 2022 @05:49AM (#30371720)
> The ATX specification requires that the power-good signal ("PWR_OK") go high no sooner than 100 ms after the power rails have stabilized, and remain high for 16 ms after loss of AC power, and fall (to less than 0.4 V) at least 1 ms before the power rails fall out of specification (to 95% of their nominal value).

I don't think that quite works for the purpose. What you'd want is a second signal that goes low as soon as possible after loss of AC power.

My reading here is that PWR_OK going low is an indication that the PSU has stopped providing good power, and the CPU must shut down immediately, or it might miscompute something due to low voltage. At this point you absolutely don't want to do any last-minute writing, you'd be risking corruption.

What you need here is an early warning signal that you can react to while the PSU is still coasting on the internal capacitors.

Re:Apple's custom NVMes are amazingly fast – if you don't ca... by jaidan on Thursday February 17, 2022 @06:12AM (#30371859)
16ms is just longer than one AC cycle at 60Hz and less than one AC cycle at 50Hz.

I would has a guess that 16ms is the physical limit for most consumer hardware (and maybe commercial computing) to detect mains loss.

Of course there is industrial hardware that can detect quicker than this but it would add a LOT of cost for arguably little gain, or something that could be solved in another manner.

Re:Apple's custom NVMes are amazingly fast – if you don't ca... by dale_glass on Thursday February 17, 2022 @07:55AM (#30372543)
> I would has a guess that 16ms is the physical limit for most consumer hardware (and maybe commercial computing) to detect mains loss.

Doubtful. 16ms is an awfully long time these days. There's no reason why you couldn't detect power loss much sooner, given a good input signal. The concept also gets used quite often, in the form of SSRs with zero crossing detection. Those are used for dimmers.

The reason is likely related to the awful waveforms produced by some UPSes and inverters:

https://www.christidis.info/images/blog/scope_20.png

Unlike a nice sine wave, those spend a good while hovering near zero volts, so the PSU has to be able to tolerate that. Detecting loss of power sooner in this case isn't a question of cost, it's a question of that you don't have a good signal to do the detection on to start with.

Re:Apple's custom NVMes are amazingly fast – if you don't ca... by mschuster91 on Thursday February 17, 2022 @11:03AM (#30374559)
> Unlike a nice sine wave, those spend a good while hovering near zero volts, so the PSU has to be able to tolerate that.

That wave chart was atrocious. I wonder if the extra load on the DC-side caps leads to them having lower life expectancy than the ones in a PSU attached to a proper power grid?

Re:Apple's custom NVMes are amazingly fast – if you don't ca... by fps-hero on Thursday February 17, 2022 @06:09AM (#30371841)
Power OK signals are used to prevent latch ups in silicon due to power glitches. The signals will route to power management ICs to ensure a full reset with proper bringing up of the power rails on any power glitch.
Re:Apple's custom NVMes are amazingly fast – if you don't ca... by fauigerzigerk on Thursday February 17, 2022 @03:37AM (#30370820)
>But they aren't doing that. I tested it on the Mac Mini. It loses several seconds of fsync()ed data on hard shutdown.

That's unfortunate. My Mac Mini crashes every other night during sleep. I guess I'm going to have to shut it down to avoid any data corruption.

Re:Apple's custom NVMes are amazingly fast – if you don't ca... by marcan_42 on Thursday February 17, 2022 @03:40AM (#30370837)
It should be flushing the drive cache on sleep. This is mostly an issue for sudden AC power loss.
Re:Apple's custom NVMes are amazingly fast – if you don't ca... by fauigerzigerk on Thursday February 17, 2022 @04:41AM (#30371261)
Ah, thanks! That's good to know.
Re:Apple's custom NVMes are amazingly fast – if you don't ca... by sgt on Thursday February 17, 2022 @05:51AM (#30371737)
Why does it crash? Mac Minis are some of the most reliable machines on the market, in my experience. Maybe a faulty unit?
Re:Apple's custom NVMes are amazingly fast – if you don't ca... by salamandersauce on Thursday February 17, 2022 @06:01AM (#30371798)
Shitty software? My 2018 Mac Mini would crash every single time going to sleep on the last version of Mojave. I'm not alone in this as there's huge threads on MacRumors and Apple's support forum about it. Apple's "fix" was to just update to Catalina which indeed fixes it but doesn't really help if you want to run 32 bit software. Wouldn't surprise me if they did something similar again.
Re:Apple's custom NVMes are amazingly fast – if you don't ca... by fauigerzigerk on Thursday February 17, 2022 @08:10AM (#30372653)
It has started crashing the night after I upgraded to macOS 12.2.0. The latest update to 12.2.1 hasn't fixed it. I'm pretty sure it's not hardware related as I had no issues before the OS upgrade.

Edit: Here's the first line of the crash log (which I'm sending to Apple every time):

  panic(cpu 3 caller 0xfffffe0023be8be0): [data.kalloc.16]: 
  element modified after free (off:0, val:0x0000000000000030, sz:16, ptr:0xfffffe2fffc9bb00)
Looks like a use after free bug.
Re:Apple's custom NVMes are amazingly fast – if you don't ca... by yborg on Thursday February 17, 2022 @12:03PM (#30375510)
This is why I'm still on Catalina. I used to be in the "trail by one point release" mode on macOS, now I'm in the "trail by 2 major releases" camp.
Re:Apple's custom NVMes are amazingly fast – if you don't ca... by david_allison on Thursday February 17, 2022 @02:49PM (#30377777)
I get regular crashes restoring from sleep on my 2014 Mac Mini (running Monterey)
Re:Apple's custom NVMes are amazingly fast – if you don't ca... by FooBarWidget on Thursday February 17, 2022 @03:48AM (#30370901)
Even on laptops I feel uncomfortable. My macOS freezes or kernel panics on me from time to time.
Re:Apple's custom NVMes are amazingly fast – if you don't ca... by marcan_42 on Thursday February 17, 2022 @03:50AM (#30370908)
I believe the NVMe driver has a kernel panic hook; I would hope it is used to issue a flush.

OTOH, if you have watchdog timeouts (I've seen this from bad drivers), those would certainly not give the kernel a chance to do that.

Re:Apple's custom NVMes are amazingly fast – if you don't ca... by ladyanita22 on Thursday February 17, 2022 @04:14AM (#30371065)
What would you implement in Asahi? Would you follow Apple's approach and defer flushes, implementing a kernel panic hook and having some kind of F_FULLFSYNC or just keep Linux' current implementation?
Re:Apple's custom NVMes are amazingly fast – if you don't ca... by marcan_42 on Thursday February 17, 2022 @04:25AM (#30371150)
We're probably going to have a knob to defer flushes (but still do them, unlike Apple, after a max timeout) that will be on by default on laptops, and make sure panics flush the cache if we can. Also apparently we need to do something for the power button too, as I just tested how macOS handles that. There is a warning before the system shuts down but we need to listen to it. Same with critical battery states.
Re:Apple's custom NVMes are amazingly fast – if you don't ca... by ladyanita22 on Thursday February 17, 2022 @04:30AM (#30371187)
Then I misunderstood. Do you mean that Apple doesn't implement ANY timeout? So they only flush when the cache is full or when a shutdown routine has started?
Re:Apple's custom NVMes are amazingly fast – if you don't ca... by marcan_42 on Thursday February 17, 2022 @04:52AM (#30371333)
They flush the cache when something requests the cache be flushed; I don't know if there is a timeout, because presumably it's not difficult for some random process to issue a FULLFSYNC and flush everything prior as a side-effect (the flush is global). But I've seen at least 5-10 seconds of data loss from drive cache loss on the Mac Mini, so if they do do deferred flushes the timeout is longer than that.
Re:Apple's custom NVMes are amazingly fast – if you don't ca... by ladyanita22 on Thursday February 17, 2022 @05:11AM (#30371461)
WTF, that is worse than I thought then. That's the dirtiest hack I've read, it's of very low quality for a company like Apple. That I'd expect for a OnePlus device, not for a full fledged Macbook.
Re:Apple's custom NVMes are amazingly fast – if you don't ca... by Dagonfly on Thursday February 17, 2022 @07:19AM (#30372287)
When do off-the-shelf NVMe controllers flush their internal DRAM buffer? I presume that happened after a timeout, even if the OS does not issue a NVMe flush command.

Does Apple implement the NVMe spec on their controller, i.e. do they indicate "Volatile Write Cache"?

Re:Apple's custom NVMes are amazingly fast – if you don't ca... by throwawaylinux on Thursday February 17, 2022 @07:45AM (#30372464)
Oh geez, deliberately issuing commands to storage after your kernel panics? It just keeps getting better :(
Re:Apple's custom NVMes are amazingly fast – if you don't ca... by coldtea on Thursday February 17, 2022 @06:24AM (#30371942)
>Perhaps real macs should be equipped with internal batteries to flush to disk in the case of power loss?

Or just add a UPS?

Re:Apple's custom NVMes are amazingly fast – if you don't ca... by dzhiurgis on Thursday February 17, 2022 @03:49AM (#30370902)
Does disk gets flushed in case of kernel panic?
Re:Apple's custom NVMes are amazingly fast – if you don't ca... by throw3838 on Thursday February 17, 2022 @05:14AM (#30371485)
How exactly this is clever? Maybe on some toy, not on workstation!