It's a specific case. Most customers won't run into it. It's to do with quantization or specifically what happens when you add enough vectors that the original clustering changes (use cases could be hit when you onboard a IM/consulting customer which has so much data to alter clusters amongst others). It is not a performance issue to be fair. Just that altering a cluster takes time.

I was just testing to see if there is a DB which does not run into such problems, before putting a business case. I am not discouraging the use, just saying that most cases do not need a vector DB. The reason we did not choose one was because our business case's number of vectors was small.

I may be misunderstanding, but I'll try to answer — quantization typically means retrieval will be slower (if referring to techniques like product quantization), but that is the case whether you're at 10K vectors or 1B vectors, afaik it doesn't really make a difference because you're only quantizing the query vector at query time (it has been awhile since I read anything on quantization, so I could be mistaken).

Maybe your question is referring to the need to have quantization at larger index sizes? In which case, yes would typically be true because you're wanting to either (1) minimize the index size when quantizing it, or (2) optimizing the query space (to search through less). Whether you want (1) or (2) will impact on the type of quantization being performed (basically 1 == product quantization, and 2 == inverted index)

So once you get to the 1M+ size, you need to consider quantization in some form - or you can go with graph-based retrieval, if you don't mind using a lot of disk space.