Milvus – An Open-Source Vector Similarity Search Engine

Re:Milvus – An Open-Source Vector Similarity Search Engine by bobosha on Friday January 10, 2020 @01:32PM (#22013603)

Great to see another ANN tool available. FAISS and SPTAG were good, but this appears to be much better. Not sure if this supports "online" learning i.e. is a training phase required?

[ Parent | on HN ]

Re:Milvus – An Open-Source Vector Similarity Search Engine by gujun720 on Friday January 10, 2020 @09:05PM (#22017554)

Please check https://medium.com/@milvusio/managing-data-in-massive-scale-...

It explains how Milvus managing vectors.

[ Parent | on HN ]

Re:Milvus – An Open-Source Vector Similarity Search Engine by ffast-math on Saturday January 11, 2020 @06:17PM (#22023110)

> "As each vector takes 2 KB space, the minimum storage space for 100 million vectors is about 200 GB"

Why are you not quantizing the vectors when you insert them? Bolt [1] and Quicker-ADC [2] make 10-100x compression basically free for approximate search (and also get you ~100x compression roughly 10x faster querying within a partition....)

[1] https://github.com/dblalock/bolt

[2] https://github.com/technicolor-research/faiss-quickeradc

[ Parent | on HN ]

Re:Milvus – An Open-Source Vector Similarity Search Engine by gujun720 on Sunday January 12, 2020 @05:34PM (#22030075)

200 GB is the size of original vectors. When creating index, Milvus supports IVF SQ8 and IVF PQ ADC.

Based on our users experience, SQ8 is the most balanced one at this moment. SQ8 provides 8x compression, higher accuracy and better performance.

[ Parent | on HN ]

Re:Milvus – An Open-Source Vector Similarity Search Engine by IfOnlyYouKnew on Friday January 10, 2020 @04:20PM (#22015657)

Yes on online learning, as I gather from the comparison at https://milvus.io/docs/v0.6.0/about_milvus/vector_db.md

[ Parent | on HN ]