Re:Ask HN: What were the things you did that made the bigges...
by ignoramous on Saturday February 01, 2020 @05:58PM
(#22213803)
In my time writing databases, I found that not only the mount-flags could result in data-loss despite fsync, given you use fsync as it is supposed to be used [0], but even enterprise-grade hardware would at times (at that scale [1]) drop and/or corrupt a few bits here and there.
That reminds me of the case where AWS famously revealed that a single NIC in their ginaromous S3 fleet flipped a single bit once in a while and that caused an outage because their gossip-daemon responsible for fleet health-checks failed spectacularly [2].
Bryan Cantrill's talk on the realted topic of hardware/firmware bugs is pretty good [3].
That reminds me of the case where AWS famously revealed that a single NIC in their ginaromous S3 fleet flipped a single bit once in a while and that caused an outage because their gossip-daemon responsible for fleet health-checks failed spectacularly [2].
Bryan Cantrill's talk on the realted topic of hardware/firmware bugs is pretty good [3].
[0] https://danluu.com/fsyncgate/
[1] https://perspectives.mvdirona.com/2017/04/at-scale-rare-even...
[2] https://youtube.com/watch?v=swQbA4zub20&t=46m02s
[3] https://youtube.com/watch?v=fE2KDzZaxvE