News for hackers. Stuff that matters.
Re:Show HN: AtomicWrite: cross platform library for atomical... by drfuchs on Tuesday March 13, 2018 @12:58AM (#16573944)
Two issues:

Your technique potentially changes the owner, group, and protection bits of the file, which may come as an unwelcome surprise to anyone trying to use it as a drop-in replacement for directly over-writing the old file. (Worse, there seems to be no way to completely fix this in Unix-land as of the last time I looked back in the 90’s, when we had to pull a feature in FrameMaker over this).

Also, be aware that (many? some?) disk controllers routinely lie about whether their internal caches are actually flushed to physical media, and there’s no way for fsync to be 100% sure that the data has really made it all the way to the spinning platter of rust. Try testing your code by putting it in a loop and repeatedly pulling the power plug out of the wall, and see if you always end up with an uncorrupted file. (Yes, people do this sort of testing on highly fault-tolerant systems.)

Re:Show HN: AtomicWrite: cross platform library for atomical... by bringtheaction on Tuesday March 13, 2018 @02:36AM (#16574207)
> Worse, there seems to be no way to completely fix this in Unix-land as of the last time I looked back in the 90’s, when we had to pull a feature in FrameMaker over this.

I had no idea FrameMaker ran on Unix. Apparently it started life on a Sun-2 workstation, and at one point according to Wikipedia, “FrameMaker ran on more than thirteen UNIX platforms, including NeXT Computer's NeXTSTEP and IBM's AIX operating systems”.

https://en.wikipedia.org/wiki/Adobe_FrameMaker

It was interesting to read how the port to Windows affected FrameMaker, and to see how long Solaris remained a supported platform.

Re:Show HN: AtomicWrite: cross platform library for atomical... by pjc50 on Tuesday March 13, 2018 @05:56AM (#16574840)
> Try testing your code by putting it in a loop and repeatedly pulling the power plug out of the wall, and see if you always end up with an uncorrupted file. (Yes, people do this sort of testing on highly fault-tolerant systems

I've had "fun" with this on some Windows CE systems, building my own little log-structured database to address the problem. All writes are appends, one disk block at a time, flushing afterwards. Worked well when using TFAT ("transactional" FAT) as the underlying file system, but when tried on a normal PC we discovered a new exciting failure mode: the file ended up with a random chunk of recently-deleted data at the end of it.

Re:Show HN: AtomicWrite: cross platform library for atomical... by matthewaveryusa on Tuesday March 13, 2018 @03:16AM (#16574332)
Is there anything you can do in software (just thinking Unix) if the hardware lies to you about fsync? Or at the very least detect such devices, or are you at the mercy of simply testing the disk?
Re:Show HN: AtomicWrite: cross platform library for atomical... by zaarn on Tuesday March 13, 2018 @03:46AM (#16574406)
You're at the mercy of the disk.

The best you can do is assume that anyone who uses your application in production will have enterprise hardware, those (usually) don't lie about fsync (consumer hdd's are somewhat likely to do this)

I've also not yet run across a SSD that lies about fsync.

The disappointing thing is that most filesystems also blindly assume this. Ext4 is somewhat robust by design but I've heard of atleast one time where a (very) cheap consumer harddisk trashed any filesystem on crash, including ZFS to the point of becoming unmountable.

Re:Show HN: AtomicWrite: cross platform library for atomical... by michaelmior on Tuesday March 13, 2018 @04:51AM (#16574611)
Did you mean lie about fsync?
Re:Show HN: AtomicWrite: cross platform library for atomical... by zaarn on Tuesday March 13, 2018 @04:54AM (#16574621)
hmm, yes, thanks.

My spelling can go bad when I type a bit fast.

Re:Show HN: AtomicWrite: cross platform library for atomical... by joetucek on Tuesday March 13, 2018 @03:45AM (#16574398)
Your OS or filesystem may maintain a blacklist of lying devices (IIRC XFS does so), although such probably isn't exported to the user. Regardless, those blacklists are built via testing, and probably aren't comprehensive. If you absolutely positively must be assured your write is persisted, I recommend fsync, sync, waiting five minutes, fsync and sync again, then top it off with a prayer.
Re:Show HN: AtomicWrite: cross platform library for atomical... by jeffreyrogers on Tuesday March 13, 2018 @10:57AM (#16576607)
Thanks, I'll look into what can be done about the owner, group, and protection bits. Obviously there's not much that can be done about the disk controllers though.