RAG at scale: Synchronizing and ingesting billions of text embeddings

Re:RAG at scale: Synchronizing and ingesting billions of tex... by ddematheu on Monday October 09, 2023 @07:26PM (#37826744)

What type of latency requirements are you dealing with? (i.e. look up time, ingestion time)

Were you using postgres already or migrated data into it?

[ Parent | on HN ]

Re:RAG at scale: Synchronizing and ingesting billions of tex... by juxtaposicion on Monday October 09, 2023 @07:57PM (#37826963)

I'd love to know the answer here too!

I've ran a few tests on pg and retrieving 100 random indices from a billion-scale table -- without vectors, just a vanilla table with an int64 primary key -- easily took 700ms on beefy GCP instances. And that was without a vector index.

Entirely possibly my take was too cursory, would love to know what latencies you're getting bryan0!

[ Parent | on HN ]

Re:RAG at scale: Synchronizing and ingesting billions of tex... by losteric on Monday October 09, 2023 @08:30PM (#37827176)

> 100 random indices from a billion-scale table -- without vectors, just a vanilla table with an int64 primary key -- easily took 700ms on beefy GCP instances.

Is there a write up of the analysis? Something seems very wrong with that taking 700ms

[ Parent | on HN ]

Re:RAG at scale: Synchronizing and ingesting billions of tex... by bryan0 on Tuesday October 10, 2023 @11:37AM (#37833496)

we have look up latency requirements on the elastic side. on pgvector it is currently a staging and aggregation database so lookup latency not so important. Our requirement right now is that we need to be able to embed and ingest ~100M vectors / day. This we can achieve without any problems now.

For future lookup queries on pgvector, we can almost always pre-filter on an index before the vector search.

yes, we use postgres pretty extensively already.

[ Parent | on HN ]