News for hackers. Stuff that matters.
Re:Investing in new vector database development vs enhancing... by gumby on Saturday November 25, 2023 @06:21PM (#38417692)
pgvector is "only" at v 0.5 but does look interesting to me. Any HNers tried it?
Re:Investing in new vector database development vs enhancing... by Arson9416 on Saturday November 25, 2023 @07:46PM (#38418228)
It's good enough and only going to get better. The postgres community only has to improve a vector index extension. These vector databases have to improve everything else (all of the really hard stuff) about a database. My money is on postgres.

EDIT>> Plus pgvector is enabled by default on AWS RDS databases. Huge bonus.

Re:Investing in new vector database development vs enhancing... by dmezzetti on Saturday November 25, 2023 @11:56PM (#38419271)
There are other approaches that could work too. For example, txtai can use Postgres as a data store and combine that with a vector index like Faiss or Hnswlib.

In this case, you get the robustness of Postgres (which has been around a while) and Faiss (which is one of the most mature vector indexes).

https://neuml.hashnode.dev/external-database-integration

Re:Investing in new vector database development vs enhancing... by threeseed on Sunday November 26, 2023 @12:09AM (#38419328)
> These vector databases have to improve everything else

No they don't. Because they aren't trying to build general purpose databases.

And often trying to jam everything into one product just makes it worse not better.

Re:Investing in new vector database development vs enhancing... by richardw on Sunday November 26, 2023 @01:03AM (#38419538)
User management, documentation, distribution, UI’s, upgrades, high availability and clustering, backups, libraries, skills required to administer and therefore hire for.

I’ve worked in companies where we grabbed any tech we want, and I’ve worked in companies with one default DB unless you had an excellent reason. The latter meant a much simpler universe where entire teams knew how to fix things that broke at 2am. You just hire one type of DBA. It also meant dev teams really knew how to squeeze the one DB platform rather than learn a little about 10 database types.

Re:Investing in new vector database development vs enhancing... by inkyoto on Sunday November 26, 2023 @12:44AM (#38419446)
> It's good enough […]

«Good enough» is not a quantative metric by any conceivable measure. Furthermore, people in surrounding threads have reported inadequate performance of the pg_vector PgSQL extension (which, indubitably, can and will be fixed at some point).

An example of a useful quantative metric would: «compared to alternative vector processing/indexing engines/databases A, B and C and a dataset comprising N inputs and a total size of S Mb/Gb/Tb and using the M, N and P use cases in the performance testing scenario and pg_vector being the performance baseline, the following results have been obtained: X, Y and Z» – do you have anything similar to that you could share? Not only the comparison results would be insightful, they could also feed into someone's decision making process of whether X should be favoured over Y.

> […] and only going to get better.

Beyond doubt, it will. In the future. But people need to solve problems in the present.

Re:Investing in new vector database development vs enhancing... by sberens on Saturday November 25, 2023 @06:26PM (#38417738)
it's significantly slower than dedicated vector dbs at >100k vectors
Re:Investing in new vector database development vs enhancing... by alex_suzuki on Sunday November 26, 2023 @03:36AM (#38420110)
I am using it successfully for doing semantic search in a database of ~1M video clips. The CLIP embeddings of key frames are stored in Postgres/pgvector and it has worked well so far.
Re:Investing in new vector database development vs enhancing... by anonzzzies on Sunday November 26, 2023 @05:17AM (#38420539)
It is not very fast (yet). On bigger sets it grinds to a halt.