Ask HN: Disk-backed NumPy for data analysis?

Re:Ask HN: Disk-backed NumPy for data analysis? by PaulHoule on Thursday February 16, 2023 @11:38AM (#34821006)

The obvious way to do a similarity query is to do a full-scan which can be done in a straightforward way against data on disk if the data is appropriately packaged. It may be slow but it works.

There are n-dimensional indexes that can accelerate similarity queries, pinecone uses them, they are not as effective as 1-d and 2-d (geospatial) indexes but they do help.

Sparse similarity search is its own problem, solved by full-text search engines like Lucene.

Dimensional reduction, say going from 300K features to 100 features, might improve the quality of your results as well as compactifying your data. It could be something you do once on a monster machine and then do many lookups on a smaller machine. That dimensional reduction might take a lot of resources, see

https://scikit-learn.org/stable/modules/decomposition.html

but in your case you might get embarassingly good results with random projections

https://en.wikipedia.org/wiki/Random_projection

[ Parent | on HN ]

Re:Ask HN: Disk-backed NumPy for data analysis? by vsroy on Thursday February 16, 2023 @12:38PM (#34822097)

Thanks for the response.

One thing to clarify: It's a 300K x 300K similarity matrix, which means I have 300K embeddings. Each embedding itself only has dimension 512.

In other words, the similarity matrix is the similarity between each embedding & every other embedding in the 300K set of embeddings.

Regardless, I think Dask will be useful here.

[ Parent | on HN ]