Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on Ask HN: Disk-backed NumPy for data analysis?. Parent comment
One thing to clarify: It's a 300K x 300K similarity matrix, which means I have 300K embeddings. Each embedding itself only has dimension 512.
In other words, the similarity matrix is the similarity between each embedding & every other embedding in the 300K set of embeddings.
Regardless, I think Dask will be useful here.
There are no comments on this comment yet.
One thing to clarify: It's a 300K x 300K similarity matrix, which means I have 300K embeddings. Each embedding itself only has dimension 512.
In other words, the similarity matrix is the similarity between each embedding & every other embedding in the 300K set of embeddings.
Regardless, I think Dask will be useful here.