Re:When your data doesn’t fit in memory: the basic techniques
by roter on Monday November 11, 2019 @05:47PM
(#21509751)
netCDF, used for storing e.g. large multidimensional climate datasets, is also "chunked" through its HDF[0] backbone.
The hard part for me is the "transpose" or "striding" problem. i.e. when the data is stored in a series of (x, y, z) files for a given hour, day, or month and I need a time series at a point.
The hard part for me is the "transpose" or "striding" problem. i.e. when the data is stored in a series of (x, y, z) files for a given hour, day, or month and I need a time series at a point.
[0] https://en.wikipedia.org/wiki/Hierarchical_Data_Format