News for hackers. Stuff that matters.
Re:Files Are Hard by notacoward on Sunday December 13, 2015 @08:38AM (#10726305)
Pretty much spot on. Local in-kernel file systems are hard, partly because of their essential nature and partly because of their history. A lot of the codebases involved still show their origins on single-core systems and pre-NCQ SATA disks, and the development/testing methods are from the same era. The developers always have time to improve the numbers on some ancient micro-benchmark, but new features often get pushed to the LVM layer (snapshots), languish for ages (unions/overlays), or are simply ignored (anything alternative to the fsync sledgehammer).

The only way a distributed file system such as I work on can provide sane behavior and decent performance to our users is to use local file systems only for course-grain space allocation and caching. Sometimes those magic incantations from ten-year-old LKML posts don't really work, because they were never really tested for more than a couple of simple cases. Other times they have unexpected impacts on performance or space consumption. Usually it's easier and/or safer just to do as much as possible ourselves. Databases - both local and distributed - are in pretty much the same boat.

Some day, I hope, all of this scattered and repeated effort will be combined into a common library that Does All The Right Things (which change over time) and adds features with a common API. It's not quite as good as if the stuff in the kernel had been done right, but I think it's the best we can hope for at this point.

Re:Files Are Hard by blakesmith on Sunday December 13, 2015 @07:26PM (#10728645)
What distributed filesystem do you work on?
Re:Files Are Hard by notacoward on Sunday December 13, 2015 @07:42PM (#10728693)
I'm on the Gluster team.
Re:Files Are Hard by dman on Tuesday December 15, 2015 @01:11PM (#10739336)
Could you provide any recommendations on reading materials for

1. File systems (single machine)

2. Distributed file systems