Re:A gentle introduction to vector databases
by dontreact on Wednesday February 23, 2022 @11:20AM
(#30442739)
Yes but my sense it is coming pretty soon for ES, and switching to Pinecone has a huge rewrite cost. As NNs keep getting better, vector search keeps becoming more important, and it becomes more and more existential for elasticsearch to do this well. I wonder if there actually some fundamental reason they won't be able to do so.
Re:A gentle introduction to vector databases
by gk1 on Wednesday February 23, 2022 @11:27AM
(#30442815)
We've had people kicking down our doors (figuratively) for the filtering feature. I have no doubt ES is getting the same feedback and is working on it.
> I wonder if there actually some fundamental reason they won't be able to do so.
This is where it gets interesting. I don't think there's anything a company with ES's resources can't do if they really want to. However...
ES's vector search is based on Lucene 9.0, which in turn uses the HNSW vector index. HNSW is shaky when it comes to CRUD updates (namely deletions), and flat out does not support metadata filtering: https://issues.apache.org/jira/browse/LUCENE-10040
Even with ES resources, it's a fundamentally hard problem. In the best case it's going to take a while to implement.
Re:A gentle introduction to vector databases
by peterstjohn on Wednesday February 23, 2022 @02:21PM
(#30445121)
If you control the HNSW implementation, it can definitely do pre-filtering. Vespa does it, and you can modify open source HNSW libs easily. I added pre-filtering support to an internal fork of HNSWLIB last week, for example…
And you're not alone... Lots of people want this, and lots of people can't get it with ES.