Re:BTables: A fast, compact format for Machine Learning
by andrewberls on Friday October 02, 2015 @09:14PM
(#10322200)
(I'm the author of the library) I'm almost certain you're correct - we (thought) we had compression enabled on our feature builder and never found the root cause, but regardless we're happy with how BTables ended up for the other reasons detailed. For future use cases we'll definitely be re-evaluating HDF5!
Re:BTables: A fast, compact format for Machine Learning
by transcranial on Friday October 02, 2015 @09:24PM
(#10322230)
HDF5 is actually a pleasure to work with in python due to h5py - and it's quite an efficient data format.
Re:BTables: A fast, compact format for Machine Learning
by RazvanPanda on Saturday October 03, 2015 @08:37AM
(#10323625)
Another interesting HDF wrapper in Rust: https://github.com/aldanor/hdf5-rs
Let me add that the OP's experience that HDF5 files were less space efficient than comparable CSV files suggest that something was grossly amiss in his use of HDF5.