News for hackers. Stuff that matters.

A comment on A Comprehensive Guide for Building Rag-Based LLM Applications. Parent comment

Re:A Comprehensive Guide for Building Rag-Based LLM Applicat... by azmodeus on Thursday September 14, 2023 @07:25AM (#37507497)
I think unless you need a vector db definitely don't use it.

A vector storage could help in reduce the time it takes to retrieve the most similar hit. I used faiss as a local vector store quite a bit to retrieve vectors fast. Though I had 1.5 million vectors to work through.

[ Parent | on HN ]
Re:A Comprehensive Guide for Building Rag-Based LLM Applicat... by chandureddyvari on Thursday September 14, 2023 @07:27AM (#37507516)
Interesting. I thought anything >1million would need a vector db to scale on production. What was your machine config for running faiss? Also did you plan for redundancy or was it just faiss as a service VM?
[ Parent | on HN ]
Re:A Comprehensive Guide for Building Rag-Based LLM Applicat... by Tostino on Thursday September 14, 2023 @06:52PM (#37515865)
People seem to underestimate the scale you can get to on a single machine, and overestimate how easy it will be to go up from there.

An in memory index is about as good as it gets for a single node performance, and fitting that many vectors into memory on a single machine is easy.

[ Parent | on HN ]