News for hackers. Stuff that matters.
Re:Moving away from HDF5 by mynewtb on Thursday January 07, 2016 @05:05PM (#10861174)
The submission talks about terabytes of data. Copying/transforming is not viable in such situations.
Re:Moving away from HDF5 by ycmbntrthrwaway on Thursday January 07, 2016 @05:16PM (#10861242)
With h5dump you can specify which datasets you want to dump. I am sure nobody is going to use awk, grep and wc to process terabytes of data. As for sanity checks, like checking that probabilities stored in a dataset sum up to 1.0 and things like that, dumping one dataset and processing it with awk should be ok.
Re:Moving away from HDF5 by paulmd on Thursday January 07, 2016 @06:17PM (#10861576)
Actually, piping data between standard Unix tools can be extremely efficient. This example is working on gigabytes rather than terabytes of data, but memory utilization is basically limited to just buffers, so you could definitely scale that to terabytes without a problem.

http://aadrake.com/command-line-tools-can-be-235x-faster-tha...