News for hackers. Stuff that matters.
Re:Moving away from HDF5 by bhouston on Thursday January 07, 2016 @03:55PM (#10860679)
Endianness generally is fast to convert, just pick a standard for the file format and detect on the platform. You can convert endianness with just shifts in C if you need to or with assembly instructions that I believe are one tick on most platforms. It is much much faster to convert endianness (again at least 10x faster) than to parse text.

It looks like you can just use these functions to convert between endianness conditionally based on the platform if you are in C/C++:

http://stackoverflow.com/a/8671129

And boost has one here: http://www.boost.org/doc/libs/1_58_0/libs/endian/doc/index.h...

Re:Moving away from HDF5 by ycmbntrthrwaway on Thursday January 07, 2016 @04:20PM (#10860872)
It is right that converting endianness is faster than reading text, but it still requires more code than calling printf and scanf. I am not into big data really and my program spends most of the time computing instead of reading and writing so 10x speedup in I/O is just not worth it. As I said, HDF5 was an overkill in my case.
Re:Moving away from HDF5 by mturmon on Thursday January 07, 2016 @08:34PM (#10862291)
The OP, and any other file format spec, needs to be able to read/write large numeric datasets. Printf is very poorly suited to this case.

Your comment is true for your use case, but it's not really responsive to the technical issue here.

Re:Moving away from HDF5 by Dylan16807 on Thursday January 07, 2016 @10:36PM (#10862763)

  for (i=start; i<end; i++) arr[i] = ntohl(arr[i]);
That's not more code than calling printf/scanf.