News for hackers. Stuff that matters.
Re:Linux's fsync() woes are getting some attention by rosser on Monday March 10, 2014 @11:28PM (#7377238)
You can always use other threads, and on most systems, you can do lots of useful I/O with reasonable performance while an fsync() is going on.

No, you can't. At least not without knowing what you're doing and careful planning.

My day job is PostgreSQL DBA, and I've been doing that for most of a decade now. As the kids on the Reddits would say, "I've seen some shit." I have some rather large servers, with some rather powerful IO subsystems — my production environment has SAS SLC SSDs under hardware RAID with a ginormous cache. I still see the behavior described in TFA far more often than I'd like. Linux really is pretty dumb here.

For example, because of this fsync() issue, and the fact that fsync() calls flush all outstanding writes for the entire filesystem the file(s) being fsync()'ed reside upon, I've set up my servers such that my $PGDATA/pg_xlog directory is a symlink from the volume mounted at (well, above) $PGDATA to a separate, much smaller filesystem. (That is: transaction logs, which must be fsync()'ed often to guarantee consistency, and enable crash recovery, reside on a smaller, dedicated filesystem, separate from the rest of my database's disk footprint.)

If I didn't do that, at every checkpoint, my performance would measurably fall. I learned this lesson the hard way, at an old job, where my postgres clusters lived on a SAN — it wasn't just my db instances that were being adversely affected by this IO storm. It was everything else that lived on the filer, too.

That's how bad it can be.

Re:Linux's fsync() woes are getting some attention by tytso on Tuesday March 11, 2014 @01:21AM (#7377590)
It's not true that fsync() calls flush all outstanding writes for the entire file system; that was true for ext3 in data=ordered mode, but it's definitely not true for ext4 or xfs. If you use fdatasync(), and there were no write commands that issued against the file descriptor that required metadata updates (i.e., you didn't do any block allocations, etc), then both ext4 and xfs won't need to trigger a journal commit, so the only thing that has to get sent to disk is all of the dirty metadata blocks, followed by a SYNC CACHE command which forces the disk drive to guarantee that all writes sent to the disk will survive a power cut.

If you use fsync() and/or you have allocated blocks or otherwise performanced a write which required updating file system metadata, and thus will require a journal commit, then you will need to force out all pending metadata updates to the journal as part of the file system commit, but that's still not the same as "flush all outstanding writes for the entire file system".

Re:Linux's fsync() woes are getting some attention by bodyfour on Tuesday March 11, 2014 @01:42AM (#7377636)
> or you have allocated blocks or otherwise performanced a write which required updating file system metadata

What if you're appending to a file, and want to checkpoint every so often. I guess you can be clever with fallocate(FALLOC_FL_KEEP_SIZE) to avoid the block allocation, but won't st_size still need to be updated?

I also assume that st_mtime doesn't count towards dirtying the metadata.

Re:Linux's fsync() woes are getting some attention by dap on Monday March 10, 2014 @11:57PM (#7377333)
You're absolutely right. The situation on Linux sounds bad, and it sounds like work's being done to improve that. Some other systems handle this case fine, though, and the problem isn't with the fsync() interface.

Aside: I worked on one of those filers, and I've seen some shit too. :) Firmware, especially disk firmware, was the worst.