Re:Moving away from HDF5
by ycmbntrthrwaway on Thursday January 07, 2016 @03:35PM
(#10860557)
The problem with binary storage is endianess. If you want portable format, you either have to use text, specify endianess in the format specification or store endianess in the file. For highest speed you need to convert data to your endianess before first use, update information about endianess inside the file and then work with memory-mapped file.
[ Parent
| on HN ]
Endianness generally is fast to convert, just pick a standard for the file format and detect on the platform. You can convert endianness with just shifts in C if you need to or with assembly instructions that I believe are one tick on most platforms. It is much much faster to convert endianness (again at least 10x faster) than to parse text.
It looks like you can just use these functions to convert between endianness conditionally based on the platform if you are in C/C++:
http://stackoverflow.com/a/8671129
And boost has one here: http://www.boost.org/doc/libs/1_58_0/libs/endian/doc/index.h...
Re:Moving away from HDF5
by ycmbntrthrwaway on Thursday January 07, 2016 @04:20PM
(#10860872)
It is right that converting endianness is faster than reading text, but it still requires more code than calling printf and scanf. I am not into big data really and my program spends most of the time computing instead of reading and writing so 10x speedup in I/O is just not worth it. As I said, HDF5 was an overkill in my case.
The OP, and any other file format spec, needs to be able to read/write large numeric datasets. Printf is very poorly suited to this case.
Your comment is true for your use case, but it's not really responsive to the technical issue here.
Re:Moving away from HDF5
by Dylan16807 on Thursday January 07, 2016 @10:36PM
(#10862763)
for (i=start; i<end; i++) arr[i] = ntohl(arr[i]);
That's not more code than calling printf/scanf.What big endian platforms do you have to support? Got an SGI Indigo somewhere in your lab?
Exactly. All the major processors are little endian now. Just standardize on little endian. If you are working on some esoteric platform that is big endian you will know it
IEEE 754 floats don't have different endianess. And as the other comment says, swapping endianess is fast (if ever needed, since most platforms are little endian or hybrid).
Re:Moving away from HDF5
by ycmbntrthrwaway on Thursday January 07, 2016 @04:26PM
(#10860912)
> IEEE 754 floats don't have different endianess.
The standard does not specify endianness [1]. From the standard point of view float is a sequence of bits. But if you memory-map the file and store float on little-endian machine, you get bytes reversed compared to how it is written in the standard. On big-endian machine sign is stored in the first byte, and on little-endian machine sign is stored in the last byte.