Choosing vector database: a side-by-side comparison

Re:Choosing vector database: a side-by-side comparison by panarky on Wednesday October 04, 2023 @12:24PM (#37767493)

I'd love to know how vector databases compare in their ability to do hybrid queries, vector similarity filtered by metadata values. For example, find the 100 items with the closest cosine similarity where genre = jazz and publication date between 1990 and 2000.

Can the vector index operate on a subset of records? Or when searching for 100 closest matches does the database have to find 1000 matches and then apply the metadata filter, and hope that doesn't reduce the result set down to zero and exclude relevant vectors?

It seems like measuring precision and recall for hybrid queries would be illuminating.

[ Parent | on HN ]

Re:Choosing vector database: a side-by-side comparison by mistrial9 on Wednesday October 04, 2023 @12:28PM (#37767570)

> do hybrid queries

"no" - the graph objects after training are opaque AFAIK

[ Parent | on HN ]

Re:Choosing vector database: a side-by-side comparison by hobs on Wednesday October 04, 2023 @01:34PM (#37768711)

Actually a lot of the databases offer filtering before or after similarity search.

[ Parent | on HN ]

Re:Choosing vector database: a side-by-side comparison by esafak on Wednesday October 04, 2023 @02:44PM (#37769746)

I'd say it's table stakes today.

[ Parent | on HN ]

Re:Choosing vector database: a side-by-side comparison by andre-z on Wednesday October 04, 2023 @12:43PM (#37767850)

There is on-stage filtering approach with extended HNSW https://qdrant.tech/articles/filtrable-hnsw/

[ Parent | on HN ]

Re:Choosing vector database: a side-by-side comparison by mvcalder on Wednesday October 04, 2023 @03:09PM (#37770083)

I can't speak to the others, but pgvector indices can "break" hybrid queries. For example, if you select using a where clause specifying metadata (where genre = jazz) and order by distance from a vector (embedding of sound clip); if the index doesn't have a lot (or any) vectors in the sphere of the query vector that also match the metadata it can return no results. I discuss this in a blog post here [1].

[1]: https://www.polyscale.ai/blog/pgvector-bigger-boat/

[ Parent | on HN ]

Re:Choosing vector database: a side-by-side comparison by gk1 on Wednesday October 04, 2023 @09:14PM (#37773811)

What you’re describing is easily done in Pinecone, and in other solutions as well. See: https://docs.pinecone.io/docs/metadata-filtering

[ Parent | on HN ]

Re:Choosing vector database: a side-by-side comparison by prabhatjha on Wednesday October 04, 2023 @11:20PM (#37774586)

You can totally do this in Cassandra. See https://docs.datastax.com/en/astra-serverless/docs/vector-se...

[ Parent | on HN ]