Once you have the embeddings in some space, for PoC I’ve mostly seen people shove them into faiss, which handles most of the rest very well for small/medium datasets: https://github.com/facebookresearch/faiss
For storing the vectors and doing the vector search: https://github.com/pgvector/pgvector
`ankane/pgvector` docker image is a drop in replacement for the postgres image, so you can fire this up with docker very quickly.
It's a normal postgres db with a vector datatype. It can index the vectors and allows efficient retrieval. Both AWS RDS and Google Cloud now support this in their managed Postgres offerings, so postgres+pgvector is a viable managed production vectordb solution.
> Also, how granular should the text chunks be?
That depends on the use case, the size of your corpus, the context of the model you are using, how much money you are willing to spend.
> Has anyone been able to achieve reliable results from these? Preferably w/o using Langchain.
Definitely. We use postgres+pgvector with php.
Interesting. Can someone explain to me why 384 specifically? How did they arrive at that number?
Different models/architectures will produce different dimension embeddings.
Here is a guide/notebook using the model above, pgvector, and a python client: https://supabase.com/docs/guides/ai/quickstarts/text-dedupli...
I think I’d need to fine tune the model to see better results with some domain-specific terms, but I couldn’t find much information about how to actually do that - what sort of input data you need, how much of it, etc.
Would be interested to hear if anyone had more to share about fine tuning these models for semantic search.
Open source and the way it works is that you give an instruction on the type of task or even domain you want the embedding to be tailored to and embeddings change depending on the instructions.
All you have to do is chunk... I'd start at a paragraph and experiment.
I would therefore suggest a better default for this, such as BERT+Qdrant.
It would be wonderful if there were a simpler (single file, SQLite or DuckDB like) database for vectors than the complex (and in some cases, unfortunately cloud-based) ones available now.
> It would be wonderful if there were a simpler (single file, SQLite or DuckDB like) database for vectors than the complex (and in some cases, unfortunately cloud-based) ones available now.
This is literally chroma!
pip install milvusMilvus will perform vector similarity search for you - all you need to do is give it a query vector.
One way to potentially answer your question about text-chunk-granularity is to take a random sample of 500 pieces of chunked text and look at several "most similar pairs." Do this for a few different chunk-lengths and you'll see how much information is lost...
https://jillesvangurp.github.io/kt-search/manual/KnnSearch.h...
You index into an index with a dense vector field
And when you query, you generate embeddings for your query and run a knn vector similarity search.
It uses some embeddings I generated with openai. You could use something like easybert or one of the many OSS embeddings models instead. Basically you need some code that converts your text/images/whatever into lists of numbers using such a model.
So:
1) use some magical tool that given a thing returns embeddings. You use this to extract embeddings at index time from your content and at query time for your queries.
2) put your embeddings along with your things in an Elasticsearch index (or vector db of your choice, Opensearch works similar to Elasticsearch for this)
3) when querying, create embeddings for your queries and find the nearest match.
I built this tutorial as a quick POC to figure out how easy it is with my own library. I'm not an expert. Mission accomplished and it only took me a few hours. The results are not impressive as this model is probably not very appropriate for the demo content. But it vaguely works. There are a bunch of people that are smarter than me that suggest that most oss models struggle to outperform bm25, which is just doing simple text searches.
Btw. the embeddings are the hard part. The rest is just plumbing. And of course world + dog just glosses over that. There's an interesting article that I came across recently that goes a bit more in depth on this: https://blog.metarank.ai/from-zero-to-semantic-search-embedd...
- Qdrant for store/inference https://qdrant.tech/documentation/tutorials/search-beginners...
Also, here's a benchmark that allows you to easily test their performance differences through a user-friendly interface. This includes both cloud solutions and open-source options. If you prefer to view pre-tested results, there are standard ones available as well. Check it out here: VectorDBBench. https://github.com/zilliztech/VectorDBBench
Video: https://www.youtube.com/watch?v=hGRNcftpqAk Slides: https://presentations.clickhouse.com/meetup74/ai/
- calculating text embeddings using open-source/local methods (not OpenAI)
- storing them in a vector database. I'm confused by the myriad of options like Chromadb, Pinecone, etc.
- running vector similarity search using open-source/local methods.
Also, how granular should the text chunks be? Too short and we'll end up with a huge database, too long and we'll probably miss some relevant information in some chunks.
Has anyone been able to achieve reliable results from these? Preferably w/o using Langchain.