Re:Loading NumPy arrays from disk: mmap() vs. Zarr/HDF5
by jmuhlich on Wednesday February 26, 2020 @09:02PM
(#22429674)
Hi Andrew! Looking into similar requirements I came across Feather and fst. They both basically let you efficiently slice into compressed on-disk DataFrames. Feather already supports Python, but fst is just for C++ and R at the moment.
https://blog.rstudio.com/2016/03/29/feather/
http://www.fstpackage.org/ https://github.com/fstpackage/fstlib
Re:Loading NumPy arrays from disk: mmap() vs. Zarr/HDF5
by dalke on Thursday February 27, 2020 @12:50AM
(#22430891)
Thanks Jeremy. I looked at Feature, and the underlying Arrow format, but couldn't figure out how to implement a row-id-label-to-row-index dictionary. The only "dictionary" I saw was for categorical data.
I had the same issue with fstlib - how do I handle id lookups?
Any pointers?
Re:Loading NumPy arrays from disk: mmap() vs. Zarr/HDF5
by jmuhlich on Friday February 28, 2020 @04:54PM
(#22446776)
Sorry, I thought Feather supported random row access but it turns out it only supports random column access.
For fst, I only played with the R interface, which would be called like this to retrieve row 12345 from the "fingerprint" column:
read_fst("library.fst", columns="fingerprint", from=12345, to=12345)
However fst didn't offer a raw/binary column datatype last I checked, which is frustrating. It has chr (string) but R can't have embedded NUL bytes in strings, so that was a dead-end for efficient storage of binary fingerprints for R. I didn't check if the underlying fstlib structures accept NULs in string columns.
Re:Loading NumPy arrays from disk: mmap() vs. Zarr/HDF5
by dalke on Tuesday March 03, 2020 @03:48AM
(#22472905)
Thanks for confirming that. Looks like I'll need to look elsewhere for the next chemfp fingerprint file format.
Re:Loading NumPy arrays from disk: mmap() vs. Zarr/HDF5
by jdnier on Thursday February 27, 2020 @12:06AM
(#22430675)
Latest info on Feather and related pyarrow:
http://arrow.apache.org/docs/python/ipc.html#feather-format
Re:Loading NumPy arrays from disk: mmap() vs. Zarr/HDF5
by Shelnutt2 on Thursday February 27, 2020 @04:53PM
(#22438358)
TileDB can handle most of your requirements today and we are working on the "row id labels" which we call axes labels[1]. We are looking to get axes labels support completed in the next 2-3 months. TileDB supports fixed size and variable length attributes so you have flexibility in how to handle your 256 byte and UTF-8 fields. On the distributed computing front, TileDB has integrations with Dask[2] and Spark[3]. We are happy to answer any further questions you might have on TileDB or how to adapt it to your use case.
[1] https://feedback.tiledb.com/tiledb-core/p/support-axes-label...
[2] https://docs.tiledb.com/developer/dask/quickstart
[3] https://docs.tiledb.com/developer/spark/quickstart
Disclosure: I am a member of the TileDB team.
I can't figure out if one of those three methods can do what I want. I want to include a multi-valued lookup table so given a string id I can find the 0 or more corresponding records. (I can handle single-valued lookup tables, if multi-valued isn't available.)
I looked at Parquet, but it doesn't seem to handle id->index table lookups. Or 256 byte fields. Do any of Zarr, HDF5 and TileDB offer a better fit?
Background: it's hard to compare molecules directly. Instead, there are hashing methods to turn a molecular structure into a fixed-length bitstring of length nearly always between 166 and 4096 bits in length.
Bitstrings are easy to compare using Jaccard similarity, which gives a decent-enough and very fast way to compare molecular similarity. With help from another HN user (who I paid), we were able to get the performance to within a factor of 2 of the theoretical memory bandwidth.
Going faster requires compressed RAM->uncompressed cache transfer, better indexing, and distributed computing. I don't know how to do it using one of the existing Big Data formats, and developing my own format is hard.
My FPB file format is a FourCC format with several chunks. One chunk stores all N fingerprints, cache size aligned. These are ordered by popcount, so in principle some should be easily compressible - I haven't tried, and can't figure out how BLOSC is supposed to be used.
Another chunk stores the corresponding fingerprint identifiers, UTF-8/non-fixed-length strings, in the same order (fingerprint[i] corresponds to id[i]).
A third chunk stores a multi-valued hash table so I can look up fingerprint(s) by name, instead of by index. The hash table is a variant of the djb's "cdb" hash table. This is a 32-bit hash, which means for large data sets I get many collisions. (I designed it for <100M entries. Some people now want to handle >1B entries.)
I'm hoping to use an existing format rather than roll my own. Again.