News for hackers. Stuff that matters.

A comment on Using the linear distance operator in Postgres 12 to find the closest match. Parent comment

Re:Using the linear distance operator in Postgres 12 to find... by phelmig on Thursday November 07, 2019 @10:45PM (#21479810)
With how many dimensions are you using Faiss with 100m+ vectors? I’m currently looking a solution to handle 1024 dimensions for ~100m items.
[ Parent | on HN ]
Re:Using the linear distance operator in Postgres 12 to find... by kaivi on Thursday November 07, 2019 @11:32PM (#21479990)
On one index I'm using OPQ16_64,IVF262144_HNSW32,PQ16 with 128 dimensions initially.

1024 dimensions is a lot! Could you elaborate on what application requires that many? If it's a DNN layer output, your data must be sparse, so dimensionality reduction won't affect your recall if tuned properly.

[ Parent | on HN ]
Re:Using the linear distance operator in Postgres 12 to find... by phelmig on Sunday November 10, 2019 @10:46AM (#21498408)
It's actually a DNN layer output. I haven't considered dimensionality reduction, yet. Thanks for pointing my there, I'll look into it. Probably thats the better way to go.

Thanks a lot for your reply!

[ Parent | on HN ]