News for hackers. Stuff that matters.
Re:Advanced Scientific Data Format by rpep on Wednesday October 05, 2022 @04:09AM (#33092662)
There's quite a bit of magic hiding under this. For e.g., try creating a NumPy array with different orderings, and the metadata looks the same:

import asdf

import numpy as np

x = np.array([[1, 2], [3, 4]], order="C")

y = np.array([[1, 2], [3, 4]], order="F")

tree = {"x": x, "y": y}

af = asdf.AsdfFile(tree)

af.write_to("example.asdf")

and you get in the metadata no distinction between the two arrays even though things like byteorder are included:

x: !core/ndarray-1.0.0

  source: 2

  datatype: int64

  byteorder: little

  shape: [2, 2]
y: !core/ndarray-1.0.0

  source: 0

  datatype: int64

  byteorder: little

  shape: [2, 2]
This makes me wonder what it's actually storing - is it actually doing something like pickling the NumPy array?
Re:Advanced Scientific Data Format by analog31 on Wednesday October 05, 2022 @08:27AM (#33094560)
I looked into this, because my team is considering using this format. I think it's storing array data as-is in straight binary. The type, order, and shape indicators are enough to tell how to recover the binary data.
Re:Advanced Scientific Data Format by rpep on Wednesday October 05, 2022 @08:39AM (#33094685)
> I think it's storing array data as-is in straight binary.

You can't infer the stride from the raw binary; a 2-D array [[1, 2], [3, 4]] in C ordering just looks like:

1,2,3,4

and in Fortran ordering it's

1,3,2,4

So there must be additional metadata stored. Maybe it's just using the .npy format internally - https://numpy.org/devdocs/reference/generated/numpy.lib.form...

Re:Advanced Scientific Data Format by throwaway29103 on Wednesday October 05, 2022 @10:41AM (#33096166)
byteorder != row/column order