Good example. definitely a problem. but that limitation exists now, so the programer would to work around it. hopefully journalling support will appear soon.
Re:Moving away from HDF5
by ipunchghosts on Thursday January 07, 2016 @06:51PM
(#10861775)
Agreed, HDF could benefit from journaling. I have to ask though, why not just make a cached file of your data and onces its done, integrate into the final HDF file?
Re:Moving away from HDF5
by ipunchghosts on Thursday January 07, 2016 @06:59PM
(#10861835)
This is true, but anything can happen while futzing with a file.
Our method is to make a copy of the file with a .tmp extension, make your mods, then rename the file and delete the old one.
HDF5 was never intended be be a container used to stream data into -- it was meant for sharing data.
I think it's reasonable to be very upset if you have a container file and adding new named chunks to the file has the possibility of causing the old data to become unreadable. It's fair in a crash before the file was saved that new chunks might be bad, but old chunks should be fine.