I looked into this, because my team is considering using this format. I think it's storing array data as-is in straight binary. The type, order, and shape indicators are enough to tell how to recover the binary data.
> I think it's storing array data as-is in straight binary.
You can't infer the stride from the raw binary; a 2-D array [[1, 2], [3, 4]] in C ordering just looks like:
1,2,3,4
and in Fortran ordering it's
1,3,2,4
So there must be additional metadata stored. Maybe it's just using the .npy format internally - https://numpy.org/devdocs/reference/generated/numpy.lib.form...
Re:Advanced Scientific Data Format
by throwaway29103 on Wednesday October 05, 2022 @10:41AM
(#33096166)
byteorder != row/column order
import asdf
import numpy as np
x = np.array([[1, 2], [3, 4]], order="C")
y = np.array([[1, 2], [3, 4]], order="F")
tree = {"x": x, "y": y}
af = asdf.AsdfFile(tree)
af.write_to("example.asdf")
and you get in the metadata no distinction between the two arrays even though things like byteorder are included:
x: !core/ndarray-1.0.0
y: !core/ndarray-1.0.0 This makes me wonder what it's actually storing - is it actually doing something like pickling the NumPy array?