Re:The Missing ”WHERE” Clause for Vector Search
by willvarfar on Monday September 27, 2021 @07:50AM
(#28669536)
How is what you describe _not_ just an efficiently implemented post filter with early out?
If it turns out that’s all pinecone have, then yeah, I’m gonna be disappointed. My mind is working overtime imagining prefixing vectors with their filter terms to root everything and other naive things…
Re:The Missing ”WHERE” Clause for Vector Search
by heipei on Monday September 27, 2021 @08:10AM
(#28669674)
I'd be surprised if it was anything else, given that they started with a working vector search database and then added filtering in after the fact.
Re:The Missing ”WHERE” Clause for Vector Search
by sgt101 on Monday September 27, 2021 @08:10AM
(#28669672)
You have high standards about what should be in a blog post.
Re:The Missing ”WHERE” Clause for Vector Search
by WithinReason on Monday September 27, 2021 @11:24AM
(#28671790)
And how do construct a search that can stream results in descending order of similarity, that's faster than O(N) for N values in the database?
Re:The Missing ”WHERE” Clause for Vector Search
by mnw21cam on Monday September 27, 2021 @11:41AM
(#28672056)
That's a fairly standard nearest neighbour search[0], usually made fast by using a kd-tree[1] or an R-tree[2].
[0] https://en.wikipedia.org/wiki/Nearest_neighbor_search
[1] https://en.wikipedia.org/wiki/K-d_tree
[2] https://en.wikipedia.org/wiki/R-tree
Re:The Missing ”WHERE” Clause for Vector Search
by gk1 on Monday September 27, 2021 @08:35AM
(#28669885)
No, this isn’t at all how it works.
This is not a trivial problem when dealing with vector similarity search, which Postgres does not have.
Re:The Missing ”WHERE” Clause for Vector Search
by svcrunch on Tuesday October 05, 2021 @10:20PM
(#28768438)
Very nice writeup. I have an idea how this might be implemented. Since you rightfully want to keep the details private, let me just ask:
How do you handle the case where the probed clusters contain no elements satisfying the filter criteria? Is the probe extended to additional clusters, or terminated with no results?
Postgres does the same thing for many queries if you look at the query plan.
Not really worthy of a blog post - especially one that says "wait till the next blog post to find out how it works!!".