1) Limit the maximum size of an update. This causes gradual degradation of the index, which is ordinarily dealt with through periodic consolidation — but now this this consolidation will be disabled in the main write mechanism.
2) Enable a background consolidation process which runs concurrently. At the end of the consolidation process a short write lock will need to be obtained while you replay all deletions that have occurred since consolidation started, but the time required isn’t problematic.
This fix required major surgery to the Lucy indexer core code, so it’s something that that would only be feasible if the data store itself supported it.
Disclaimer: I'm from Pinecone, hello!
1. gRPC. Using gRPC to write vectors has had a really nice performance boost. It is released in Weaviate core but here is still some work on do on the clients. Feel free to get in contact if you would like to try it out.
2. Parameter tuning. lowering `efConstruction` can speed up imports.
3. We are also working on async indexing https://github.com/weaviate/weaviate/issues/3463 which will further speed things up.
In comparison with pgvector, Weaviate has more flexible query options such as hybrid search and quantization to save memory on larger datasets.
We have around 2 million 768-length embeddings + metadata -- writing too fast to Weaviate quickly results in OOMs. We circumvented this by very slowly trickling our data through to Weaviate, but this causes many-hours-long writes.
I'm experimenting with dumping our data into postgres and building an hnsw index with pgvector once all writes are done. This has reduced our time to maybe 2 hours at most (the bulk of that being index building time, of course).