Re:Supabase Vector, the Open Source Vector Toolkit for Postgres
by brigadier132 on Thursday June 08, 2023 @11:36AM
(#36243667)
When you say you can do vss along with filtering. Does the filtering happen before or after the vss?
Re:Supabase Vector, the Open Source Vector Toolkit for Postgres
by roseway4 on Thursday June 08, 2023 @12:18PM
(#36244395)
Logically at the same time in the sql query. This may not be how the query is executed by the database, though, and is up to the optimizer.
Re:Supabase Vector, the Open Source Vector Toolkit for Postgres
by kiwicopple on Thursday June 08, 2023 @12:20PM
(#36244423)
> It’s important indexes are added after a fair number of vectors have been persisted, which can make some production use cases challenging to implement.
managing indexes are the most challenging part of using pgvector right now. There is a bit of a "goldilocks zone" - you can't create the index with zero data, but if you create it too late then it can take a lot of memory. Andrew (and a few others, like AWS) are working hard on improving this, as well as researching other index types.
Re:Supabase Vector, the Open Source Vector Toolkit for Postgres
by deepsquirrelnet on Thursday June 08, 2023 @02:49PM
(#36246734)
That’s good to know. I tested it in a docker container and indexed after loading ~1M vectors. Checking EXPLAIN showed it wasn’t using the index at all. I wonder if it’s because it couldn’t load into memory.
Without an index, I found it was too slow for data that size. I didn’t expect issues for data at this “small”, but it’s pretty new to me.
Re:Supabase Vector, the Open Source Vector Toolkit for Postgres
by sodiumSalt on Thursday June 08, 2023 @02:21PM
(#36246331)
Zep seems really cool! I think hybrid search is a great paradigm and believe it will continue to grow. How do you guys plan on dealing with data deletion requests and other GDPR rules around storage?
It’s super useful for hybrid search: vector similarity search over unstructured data joined with filtering by structured data.
One caveat: It’s important indexes are added after a fair number of vectors have been persisted, which can make some production use cases challenging to implement.
[0] https://github.com/getzep/zep