News for hackers. Stuff that matters.
Re:Moving away from HDF5 by iheartmemcache on Thursday January 07, 2016 @04:05PM (#10860746)
It's also really useful if you have a lot of numerical data streaming in that you want to store and use at a later date. CERN results I'm sure use something similar to HDF5, nearly all of HFT algo trading uses HDF5 for securities they are going to explore down the road but don't want to waste KDB+ licenses on, Google File System's chunking scheme seems to be somewhat similar to it as well. _"Third, most files are mutated by appending new data rather than overwriting existing data. ... Once written, the files are only read, and often only sequentially."_ [1] _That_ is the use case for HDF5. The problem is this guy tried to slam a circular peg into a square hole. I'm in no way an apologist for HDF5 but his complaints are terribly vague. "Limited support for parallel access" Then you go read the source[2] and and see GIL complaints abound. And again, this was meant for an append-only situation where you shouldn't even have to acquire a lock in the first place since there's no contention possibility! "Impossibility to explore datasets with standard Unix/Windows tools" right, but there are plenty of Java tools that perform quite well, even with a cold JVM. "Opacity of the development and slow reactivity of the development team." AFAIK it's an open-source project, this complaint is valid if you're paying a vendor fees for a product and have a support plan with an SLA, and it's not valid in the least otherwise. "High risks of data corruption" I've never once seen this happen when HDF5 was properly used, though I'd love to see a pdb dump of the state his program when that occurred. Open offer - I'll fix that bug if it's a fault with the C lib you're FFI'ing with.

edit: oh, the Java tooling was already mentioned.

[1] http://static.googleusercontent.com/media/research.google.co... [2] https://github.com/h5py/h5py/blob/master/h5py/_locks.pxi

Re:Moving away from HDF5 by batbomb on Thursday January 07, 2016 @04:17PM (#10860842)
CERN uses ROOT.
Re:Moving away from HDF5 by iheartmemcache on Thursday January 07, 2016 @04:42PM (#10861018)
Huh! Good to know. ftp://root.cern.ch/root/doc/11InputOutput.pdf[1] The spec for anyone interested. For comparison:https://www.hdfgroup.org/projects/hdf5_aip/aip15.gif to page 6 on CERN's PDF.
Re:Moving away from HDF5 by saboot on Friday January 08, 2016 @12:17AM (#10863082)
ROOT has been making progress with regards to one of the points addressed in the post. Regarding needing a special program to view an HDF5 file, root has developed a JavaScript viewer which can display data and stored plots and graphs. It's pretty cool, HDF5 could do something similar if it doesn't already exist. https://root.cern.ch/js/