News for hackers. Stuff that matters.
Re:Moving away from HDF5 by zvrba on Friday January 08, 2016 @02:02AM (#10863399)
In my previous job we were evaluating HDF5 for implementing a data-store a couple of years ago. We had some strict requirements about data corruption (e.g., if the program crashes amid a write operation in another thread, the old data must be left intact and readable), as well as multithreaded access. HDF5 supports parallelism from distinct programs, but its multithreaded story was (is?) very weak.

I ended up designing a transactional file format (a kind of log-structured file system with data versioning and garbage collection, all stored in a single file) from scratch to match our requirements. Works flawlessly with terabytes of data.

Re:Moving away from HDF5 by srean on Friday January 08, 2016 @03:51AM (#10863722)
Might one take a peek at that ? In other words was it open sourced or plans to that effect exists ?
Re:Moving away from HDF5 by zvrba on Friday January 08, 2016 @05:19AM (#10863971)
No and no. Strictly proprietary technology which gives a real competitive advantage. Fun thing is, if you choose your data structures wisely, it's not even that hard to write; it ended up being under 2k lines of C++ code.

GC was offline though; it was performed at the time the container was "upgraded" from RO to RW access. I don't think it'd be difficult to make it online, but there was no need for that.