A common pattern (that my scientific software used) was this: an initial file is created from the raw data pulled from the sequencer. After sequencing, you could run all sorts of analyses. Sometimes the analyses themselves, and sometimes intermediate results, where very slow to compute and hence cached in the file.
I think it's reasonable to be very upset if you have a container file and adding new named chunks to the file has the possibility of causing the old data to become unreadable. It's fair in a crash before the file was saved that new chunks might be bad, but old chunks should be fine.
Good example. definitely a problem. but that limitation exists now, so the programer would to work around it. hopefully journalling support will appear soon.
Re:Moving away from HDF5
by ipunchghosts on Thursday January 07, 2016 @06:51PM
(#10861775)
Agreed, HDF could benefit from journaling. I have to ask though, why not just make a cached file of your data and onces its done, integrate into the final HDF file?
Re:Moving away from HDF5
by ipunchghosts on Thursday January 07, 2016 @06:59PM
(#10861835)
This is true, but anything can happen while futzing with a file.
Our method is to make a copy of the file with a .tmp extension, make your mods, then rename the file and delete the old one.
HDF5 was never intended be be a container used to stream data into -- it was meant for sharing data.
Re:Moving away from HDF5
by ipunchghosts on Thursday January 07, 2016 @06:45PM
(#10861737)
Good post!
Bugs and crashes in the HDF5 library and in the wrappers - sure, every sw has bugs. But in over 15 years of using HDF, I have not seen a bug that stopped me from doing what I want. And the HDF team is very responsive in fixing / suggesting work arounds.
Poor performance in some situations - yes & no. A well built library with a well designed application should approach posix performance. But HDF is not a simple file format, so expect some overhead.
Limited support for parallel access - Parallel HDF is one of the most, if not the top most, popular library for parallel IO. Parallel HDF also uses MPI. If your app is not MPI, you cant use Parallel HDF. If the "parallel access" refers to threading, HDF has a thread safe feature that you need to enable when building the code. If "parallel access" refers to access from multiple processes, then HDF is not the right file format to use. you could do it for read-only purposes but not write. again, not the right motivation to pick HDF
Impossibility to explore datasets with standard Unix/Windows tools - again, HDF is not a flat file, so how can one expect standard tools to read it? its like saying I would like to use standard tools to explore a custom binary file format I came up with. wrong expectations.
Hard dependence on a single implementation of the library - afaik there is only one implementation of the spec. the author seems to know this before deciding on HDF. Why is this an issue if its already known?
High complexity of the specification and the implementation -
Opacity of the development and slow reactivity of the development team - slow reactivity to what? HDF source is available so one can go fix / modify whatever they want.
seems the author picked HDF with wrong assumptions.
HDF serves a huge community that has specific requirements, one of which is preserving precision, portability, parallel access, being able to read/write datasets, query the existing file for information of the data in the file, multi dimensional datasets, large amount of data to fit in a single file, etc.
https://www.hdfgroup.org/why_hdf/