Re:Similarity Search and Deduplication at Scale
by saintarian on Saturday November 12, 2022 @11:06PM
(#33579926)
Shameless plug - for folks who don't want to take on the work of model selection, on-demand scaling of model serving, scaling the vector database for search set size and query throughput, we built a service that hides all this behind a simple API [1]. The example in [1] is for images, but here is quick-start for text [2].
[1] https://www.nyckel.com/semantic-image-search [2] https://www.nyckel.com/docs/text-search-quickstart
Re:Similarity Search and Deduplication at Scale
by marginalia_nu on Sunday November 13, 2022 @06:42AM
(#33582139)
Maybe from ... "at scale" bit? ML approaches are relatively computationally expensive.
A great upside to this approach is that it works for a variety of different types of unstructured data (images, video, molecular structures, geospatial data, etc), not just text. The rise of multimodal models such as CLIP (https://openai.com/blog/clip) makes this even more relevant today. Combine it with a vector database such as Milvus (https://milvus.io) and you'll be able to do this at scale with very minimal effort.