News for hackers. Stuff that matters.
Re:Moving away from HDF5 by echlebek on Thursday January 07, 2016 @08:16PM (#10862226)
Funny enough, we started out with a file-based system like the one you described, and moved to an HDF5-based system.

Are you not concerned about i-node consumption with the file-system based approach?

Re:Moving away from HDF5 by nimrody on Friday January 08, 2016 @01:46AM (#10863348)
It depends on whether you have many small files or a small number of large files. In the former case, one file approach (HDF5) makes sense. In the latter case, you don't have a problem with i-node consumption and you gain the ability to easily access only the data you need without bringing the entire data-set into memory.

I kind of the like the approach suggested above - if you have many small files store them in a zip archive and use some library to access the data directly.