News for hackers. Stuff that matters.
Re:Moving away from HDF5 by x0x0 on Thursday January 07, 2016 @03:32PM (#10860535)
A common pattern (that my scientific software used) was this: an initial file is created from the raw data pulled from the sequencer. After sequencing, you could run all sorts of analyses. Sometimes the analyses themselves, and sometimes intermediate results, where very slow to compute and hence cached in the file.

I think it's reasonable to be very upset if you have a container file and adding new named chunks to the file has the possibility of causing the old data to become unreadable. It's fair in a crash before the file was saved that new chunks might be bad, but old chunks should be fine.

Re:Moving away from HDF5 by skynetv2 on Thursday January 07, 2016 @03:56PM (#10860686)
Good example. definitely a problem. but that limitation exists now, so the programer would to work around it. hopefully journalling support will appear soon.
Re:Moving away from HDF5 by ipunchghosts on Thursday January 07, 2016 @06:51PM (#10861775)
Agreed, HDF could benefit from journaling. I have to ask though, why not just make a cached file of your data and onces its done, integrate into the final HDF file?
Re:Moving away from HDF5 by ipunchghosts on Thursday January 07, 2016 @06:59PM (#10861835)
This is true, but anything can happen while futzing with a file.

Our method is to make a copy of the file with a .tmp extension, make your mods, then rename the file and delete the old one.

HDF5 was never intended be be a container used to stream data into -- it was meant for sharing data.