liliuleo93

Thanks for your suggestion and this is a super good question. I was asked some times and please allow me quote one of my response in the repo

" With respect to recall vs Performance, your idea is indeed correct. However, several reasons have guided us to our current approach:

1. We are not solely benchmarking open-source systems; we are also focusing on cloud services. Some of these services, such as Zilliz and Pinecone, don't allow users to customize their parameters to tune the recall, aiming to simplify their usage. Consequently, creating a recall vs Performance graph is not feasible. Also this benchmark allow users to customize their parameters for systems allowing tuning to get their own result to do comparison.

2. There already exists a number of benchmarks doing what you've suggested, which target individuals with ANN search backgrounds. Our goal is to make this benchmark as straightforward as possible and to assist people who lack understanding about the inner workings of each system.

3. Concerning reproducibility, generating a recall vs QPS graph that you mentioned, would require conducting a multitude of tests to obtain enough data points, which considerably reduces reproducibility. "

the link is: https://github.com/zilliztech/VectorDBBench/issues/200#issue...

I apologize for the repeated posts. The reason I posted it 6 times is that I aim to announce every release and significant commit. The reason my entire history is centered around this benchmark is because I wanted to introduce my project to the community, potentially with some bias. I began my Hacker News journey at that time and wanted to share what I was working on.

Fully agree with this idea. All tricks can be a real world strategy and it is impossible for anyone to claiming that they have an absolute fair benchmark.

So the only way we can do to approaching it is to provide more real-world-like cases and forget all tricks vendors might play inside their systems.

Also, people will concern the representative of the cases benchmarks provide. So we plan to make this benchmark more like a framework to support customized cases in the next step.

Yes, of course vendors have bias. But IMHO, if a benchmark is reproducible and the use cases can match users' needs, then we can say it can somehow help decision making.

Plz feel free to try the benchmark tool and make an issue with more details if it is not reproducible. Then we can dig it to see if there is any misusage or bug in the benchmark code. :)

Hi, if you're looking into vector storage and querying, there are several things to consider. For lightweight usage, you can directly use FAISS without any database overhead. For heavy usage, Milvus/Zilliz is the most production-ready solution.

Also, here's a benchmark that allows you to easily test their performance differences through a user-friendly interface. This includes both cloud solutions and open-source options. If you prefer to view pre-tested results, there are standard ones available as well. Check it out here: VectorDBBench. https://github.com/zilliztech/VectorDBBench