Re:Ask HN: What were the things you did that made the bigges...
by ignoramous on Saturday February 01, 2020 @05:58PM
(#22213803)
In my time writing databases, I found that not only the mount-flags could result in data-loss despite fsync, given you use fsync as it is supposed to be used [0], but even enterprise-grade hardware would at times (at that scale [1]) drop and/or corrupt a few bits here and there.
That reminds me of the case where AWS famously revealed that a single NIC in their ginaromous S3 fleet flipped a single bit once in a while and that caused an outage because their gossip-daemon responsible for fleet health-checks failed spectacularly [2].
Bryan Cantrill's talk on the realted topic of hardware/firmware bugs is pretty good [3].
[0] https://danluu.com/fsyncgate/
[1] https://perspectives.mvdirona.com/2017/04/at-scale-rare-even...
The other one is that getting file safety correctly is really hard. The OS likes to the developer, the filesystem lies to the OS and the hardware lies to everyone. For most applications, that's not a big deal, but for something that used to run every 15 seconds on hundreds of millions of computers, this can cause data loss. That's why you really want to use a DBMS rather than roll your own format if data safety is critical.
For shutdown, it was also a case of initial architecture not matching the current situation anymore. Firefox was initially a synchronous, single-threaded, single-process architecture, but this had stopped being the case for a few years already. At some point, shutdown needed to be re-architectured, I happened to be the one who managed to convince people that the time was now :)