I'd have thought MPI-IO is the relevant standard for HPC i/o (perhaps under something like HDF5). I'd expect HPC applications with serious i/o requirements to be using that anyway.
Re:What's So Bad About Posix I/O?
by glennklockwood on Tuesday September 12, 2017 @10:48AM
(#15227684)
It is and people do use it, but it ultimately still exists as software that sits on top of POSIX in the vast majority of cases. It can clean up really bad POSIX I/O by buffering and reordering, but doing so induces latency, memory overheads, and consumes other resources that don't always pay dividends at all scales.
ROMIO unfortunately requires global locking on Lustre, but I don't see how that's relevant to the use of standard or other high-level (HDF, NetCDF, ...) non-POSIX interfaces. You can swap in PVFS, or something else with a different architecture. (So what if object stores are on _local_ POSIX-y filesystems?)
I'd have thought the salient feature is collective i/o, not cleaning up silly POSIX i/o of some sort. Nothing's perfect, but what should replace the non-blocking collective MPI-IO under5, for instance? I wonder what are the major NERSC applications that don't do sane i/o and should. The published workload characterization doesn't help.
Applications using MPI-IO are still typically run on top of a parallel file system. The MPI-IO API is, in simple terms, intended to create I/O workloads that should perform well on top of the POSIX I/O interface (e.g. large, sequential I/O). But my understanding is that MPI-IO can have some major drawbacks, like a huge amount of compute-side network utilization used to reorganize and stage I/O.
The PFS under MPI-IO clearly doesn't have to be a POSIX one; PVFS is stateless. I can't comment on that specific intention, which isn't in the standard's rationale. Where is it documented? You'd have to argue with Rob Latham about whether "MPI is a good setting for parallel i/o".
"POSIX I/O is simply not what HPC needs." but it's ok for almost any other use case and "There are many ways to move beyond the constraints imposed by POSIX I/O without rewriting applications, and this is an area of active research in both industry and academia." The problem is the lack of a standard solution yet.
The post does a good job at explaining why HPC doesn't need the strong guarantees provided by POSIX I/O and why they slow down those kind of applications. They are perfectly fine and desirable for most workloads on single user workstations, such as web development (save a file in the editor and the compiler gets its bytes exactly as you wrote them.)