News for hackers. Stuff that matters.

A comment on Ask HN: Disk-backed NumPy for data analysis?.

Re:Ask HN: Disk-backed NumPy for data analysis? by jstx1 on Thursday February 16, 2023 @11:36AM (#34820962)
> I want to make queries on this matrix like (what are the rows most similar to row #2)

There are libraries for efficient vector similarity search like https://github.com/facebookresearch/faiss

[ Parent | on HN ]
Re:Ask HN: Disk-backed NumPy for data analysis? by kacperlukawski on Thursday February 16, 2023 @11:44AM (#34821122)
For that scale FAISS might be not enough, as it keeps the data in memory and is simply hard to scale. It's worth considering to use a proper vector database in a distributed mode to spread the load, or just allow storing some data on disk: https://qdrant.tech/articles/memory-consumption/
[ Parent | on HN ]