The schema-based serialization libraries (Thrift, Protobufs) or MsgPack are a good way to avoid that too. They come with a lot less baggage than say HDF5. Also, if efficiency isn't paramount -- just use SQLite! Amazing tool when it's in its sweetspot.
Lots of tradeoffs when dealing w/ serialization and file formats, no easy answers.
edit: oh, the Java tooling was already mentioned.
[1] http://static.googleusercontent.com/media/research.google.co... [2] https://github.com/h5py/h5py/blob/master/h5py/_locks.pxi
It looks like you can just use these functions to convert between endianness conditionally based on the platform if you are in C/C++:
http://stackoverflow.com/a/8671129
And boost has one here: http://www.boost.org/doc/libs/1_58_0/libs/endian/doc/index.h...
Your comment is true for your use case, but it's not really responsive to the technical issue here.
for (i=start; i<end; i++) arr[i] = ntohl(arr[i]);
That's not more code than calling printf/scanf.The standard does not specify endianness [1]. From the standard point of view float is a sequence of bits. But if you memory-map the file and store float on little-endian machine, you get bytes reversed compared to how it is written in the standard. On big-endian machine sign is stored in the first byte, and on little-endian machine sign is stored in the last byte.
Research data are almost always specific. The idea of a general format serving all research just sounds silly to me.
its also really easy to write a viewer in python using matplotlib.
If you want headers, you can define those as text and just use extents to embed the binary data. Tar is another alternative, pretty easy to implement.
For simple uses I now use '%a' printf specifier. It is specifically designed to avoid losing a single bit of informaiton. And you can easily read floats stored this way in numpy by using genfromtxt with optional converters= argument and builtin float.fromhex function.