News for hackers. Stuff that matters.
Re:BTables: A fast, compact format for Machine Learning by fjordster on Friday October 02, 2015 @06:57PM (#10321831)
HDF5 isn't perfect but it does this kind of job pretty well. The C, C++, and HDF5 APIs are definitely not fun to use, but there are wonderful and intuitive APIs available in some languages---I'm thinking of Python's h5py here.

Let me add that the OP's experience that HDF5 files were less space efficient than comparable CSV files suggest that something was grossly amiss in his use of HDF5.

Re:BTables: A fast, compact format for Machine Learning by andrewberls on Friday October 02, 2015 @09:14PM (#10322200)
(I'm the author of the library) I'm almost certain you're correct - we (thought) we had compression enabled on our feature builder and never found the root cause, but regardless we're happy with how BTables ended up for the other reasons detailed. For future use cases we'll definitely be re-evaluating HDF5!
Re:BTables: A fast, compact format for Machine Learning by transcranial on Friday October 02, 2015 @09:24PM (#10322230)
HDF5 is actually a pleasure to work with in python due to h5py - and it's quite an efficient data format.
Re:BTables: A fast, compact format for Machine Learning by RazvanPanda on Saturday October 03, 2015 @08:37AM (#10323625)
Another interesting HDF wrapper in Rust: https://github.com/aldanor/hdf5-rs