Pg_vectorize: The simplest way to do vector search and RAG on Postgres

Re:Pg_vectorize: The simplest way to do vector search and RA... by samier-trellis on Wednesday March 06, 2024 @11:46AM (#39617829)

Can you speak a bit to the speed/slowness of doing such chunking? We recently started using LLMs to clean the text (data quality/text cleanliness is a problem for us), and it has increased our indexing time a lot.

[ Parent | on HN ]

Re:Pg_vectorize: The simplest way to do vector search and RA... by CuriouslyC on Wednesday March 06, 2024 @12:10PM (#39618137)

It's going to depend on what you're running on, but phi2 is pretty fast so you can reasonably expect to be hitting ~50 tokens a second. Given that, if you are ingesting a 100k token document you can expect it to take 30-40 minutes if done serially, and you can of course spread stuff in parallel.

[ Parent | on HN ]

Re:Pg_vectorize: The simplest way to do vector search and RA... by samier-trellis on Wednesday March 06, 2024 @12:26PM (#39618340)

thanks for the info--good to know we aren't the only ones contending with speed for large documents lol

[ Parent | on HN ]