News for hackers. Stuff that matters.
Re:Loading NumPy arrays from disk: mmap() vs. Zarr/HDF5 by dalke on Wednesday February 26, 2020 @04:21PM (#22427473)
Given the discussion on Zarr, HDF5 and TileDB, I hope I can get feedback from a problem I have. I want to switch from a format I made (the "FPB" format described at http://chemfp.com/fpb_format/ ) to one which scales to more than 250 million entries. (See https://jcheminf.biomedcentral.com/articles/10.1186/s13321-0... for my recent publication on the system, which implements similarity search for cheminformatics fingerprints.)

I can't figure out if one of those three methods can do what I want. I want to include a multi-valued lookup table so given a string id I can find the 0 or more corresponding records. (I can handle single-valued lookup tables, if multi-valued isn't available.)

I looked at Parquet, but it doesn't seem to handle id->index table lookups. Or 256 byte fields. Do any of Zarr, HDF5 and TileDB offer a better fit?

Background: it's hard to compare molecules directly. Instead, there are hashing methods to turn a molecular structure into a fixed-length bitstring of length nearly always between 166 and 4096 bits in length.

Bitstrings are easy to compare using Jaccard similarity, which gives a decent-enough and very fast way to compare molecular similarity. With help from another HN user (who I paid), we were able to get the performance to within a factor of 2 of the theoretical memory bandwidth.

Going faster requires compressed RAM->uncompressed cache transfer, better indexing, and distributed computing. I don't know how to do it using one of the existing Big Data formats, and developing my own format is hard.

My FPB file format is a FourCC format with several chunks. One chunk stores all N fingerprints, cache size aligned. These are ordered by popcount, so in principle some should be easily compressible - I haven't tried, and can't figure out how BLOSC is supposed to be used.

Another chunk stores the corresponding fingerprint identifiers, UTF-8/non-fixed-length strings, in the same order (fingerprint[i] corresponds to id[i]).

A third chunk stores a multi-valued hash table so I can look up fingerprint(s) by name, instead of by index. The hash table is a variant of the djb's "cdb" hash table. This is a 32-bit hash, which means for large data sets I get many collisions. (I designed it for <100M entries. Some people now want to handle >1B entries.)

I'm hoping to use an existing format rather than roll my own. Again.

Re:Loading NumPy arrays from disk: mmap() vs. Zarr/HDF5 by jmuhlich on Wednesday February 26, 2020 @09:02PM (#22429674)
Hi Andrew! Looking into similar requirements I came across Feather and fst. They both basically let you efficiently slice into compressed on-disk DataFrames. Feather already supports Python, but fst is just for C++ and R at the moment.

https://blog.rstudio.com/2016/03/29/feather/

http://www.fstpackage.org/ https://github.com/fstpackage/fstlib

Re:Loading NumPy arrays from disk: mmap() vs. Zarr/HDF5 by dalke on Thursday February 27, 2020 @12:50AM (#22430891)
Thanks Jeremy. I looked at Feature, and the underlying Arrow format, but couldn't figure out how to implement a row-id-label-to-row-index dictionary. The only "dictionary" I saw was for categorical data.

I had the same issue with fstlib - how do I handle id lookups?

Any pointers?

Re:Loading NumPy arrays from disk: mmap() vs. Zarr/HDF5 by jmuhlich on Friday February 28, 2020 @04:54PM (#22446776)
Sorry, I thought Feather supported random row access but it turns out it only supports random column access.

For fst, I only played with the R interface, which would be called like this to retrieve row 12345 from the "fingerprint" column:

  read_fst("library.fst", columns="fingerprint", from=12345, to=12345)
However fst didn't offer a raw/binary column datatype last I checked, which is frustrating. It has chr (string) but R can't have embedded NUL bytes in strings, so that was a dead-end for efficient storage of binary fingerprints for R. I didn't check if the underlying fstlib structures accept NULs in string columns.
Re:Loading NumPy arrays from disk: mmap() vs. Zarr/HDF5 by dalke on Tuesday March 03, 2020 @03:48AM (#22472905)
Thanks for confirming that. Looks like I'll need to look elsewhere for the next chemfp fingerprint file format.
Re:Loading NumPy arrays from disk: mmap() vs. Zarr/HDF5 by jdnier on Thursday February 27, 2020 @12:06AM (#22430675)
Latest info on Feather and related pyarrow: http://arrow.apache.org/docs/python/ipc.html#feather-format
Re:Loading NumPy arrays from disk: mmap() vs. Zarr/HDF5 by Shelnutt2 on Thursday February 27, 2020 @04:53PM (#22438358)
TileDB can handle most of your requirements today and we are working on the "row id labels" which we call axes labels[1]. We are looking to get axes labels support completed in the next 2-3 months. TileDB supports fixed size and variable length attributes so you have flexibility in how to handle your 256 byte and UTF-8 fields. On the distributed computing front, TileDB has integrations with Dask[2] and Spark[3]. We are happy to answer any further questions you might have on TileDB or how to adapt it to your use case.

[1] https://feedback.tiledb.com/tiledb-core/p/support-axes-label...

[2] https://docs.tiledb.com/developer/dask/quickstart

[3] https://docs.tiledb.com/developer/spark/quickstart

Disclosure: I am a member of the TileDB team.