News for hackers. Stuff that matters.
Re:The Missing ”WHERE” Clause for Vector Search by londons_explore on Monday September 27, 2021 @07:44AM (#28669494)
This works by streaming results from the similarity search to the filter, in descending order of similarity. As soon as enough matches are found, the similarity search is terminated.

Postgres does the same thing for many queries if you look at the query plan.

Not really worthy of a blog post - especially one that says "wait till the next blog post to find out how it works!!".

Re:The Missing ”WHERE” Clause for Vector Search by willvarfar on Monday September 27, 2021 @07:50AM (#28669536)
How is what you describe _not_ just an efficiently implemented post filter with early out?

If it turns out that’s all pinecone have, then yeah, I’m gonna be disappointed. My mind is working overtime imagining prefixing vectors with their filter terms to root everything and other naive things…

Re:The Missing ”WHERE” Clause for Vector Search by heipei on Monday September 27, 2021 @08:10AM (#28669674)
I'd be surprised if it was anything else, given that they started with a working vector search database and then added filtering in after the fact.
Re:The Missing ”WHERE” Clause for Vector Search by sgt101 on Monday September 27, 2021 @08:10AM (#28669672)
You have high standards about what should be in a blog post.
Re:The Missing ”WHERE” Clause for Vector Search by WithinReason on Monday September 27, 2021 @11:24AM (#28671790)
And how do construct a search that can stream results in descending order of similarity, that's faster than O(N) for N values in the database?
Re:The Missing ”WHERE” Clause for Vector Search by mnw21cam on Monday September 27, 2021 @11:41AM (#28672056)
That's a fairly standard nearest neighbour search[0], usually made fast by using a kd-tree[1] or an R-tree[2].

  [0] https://en.wikipedia.org/wiki/Nearest_neighbor_search
  [1] https://en.wikipedia.org/wiki/K-d_tree
  [2] https://en.wikipedia.org/wiki/R-tree
Re:The Missing ”WHERE” Clause for Vector Search by gk1 on Monday September 27, 2021 @08:35AM (#28669885)
No, this isn’t at all how it works.

This is not a trivial problem when dealing with vector similarity search, which Postgres does not have.

Re:The Missing ”WHERE” Clause for Vector Search by svcrunch on Tuesday October 05, 2021 @10:20PM (#28768438)
Very nice writeup. I have an idea how this might be implemented. Since you rightfully want to keep the details private, let me just ask:

How do you handle the case where the probed clusters contain no elements satisfying the filter criteria? Is the probe extended to additional clusters, or terminated with no results?