jkb79

ColBER: Contextualized Late Interaction over BERT. That is just the name. It can be fine-tuned for retrieval using any encoder-only model like the ones you mention.

Yes, Vespa has a generic Tensor framework that allows you to index multiple vectors for a single field, see https://blog.vespa.ai/semantic-search-with-multi-vector-inde... for details.

field embeddings type tensor<float>(p{}, x[384]) to represent a multi-vector field { "0": [0.1....], "1": [0.2,..] }

Actually, Vespa comes out of the same FAST company. Yahoo bought Overture/Altavista and a lot of other web search companies in 2003, including the web search division of FAST. The Enterprise search division of FAST was later acquired by Microsoft.

Hehe, it was a joke, we don't have polar bears on the mainland of Norway. But, it was fun to show the photo to visitors from different countries. "Be careful when you walk back to the hotel".

Disclaimer, I work on Vespa.

If you look for just pure vector similarity search, there are many alternatives. But Vespa's tensor support, multi-vector indexing and the ability to express models like colBERT (1) or cross-encoders makes it stand out if you need to move beyond pure vector search support.

Plus, for RAG use cases, it's a full blown text search engine as well, allowing hybrid ranking combinations. Also with many pure vector databases like Pinecone, you cannot describe an object with more than one vector, if you have different vector models for the object, you need different indexes, and then duplicate metadata across those indexes (if you need filtering + vector search).

1 https://blog.vespa.ai/pretrained-transformer-language-models...

Vespa was the internal code name for the project going back to 2005ish, vertical search platform.

Thank you for the shout-out Frank!

Yeah, I worked with the Flickr team on that project. Scaling to billions of photos, with partial update support of popularity for ranking.

Back then, the properties had to stand up their own Vespa cluster(s), later on we created a managed service out of it. And, yes, the original plan for Vespa was to be a Vertical Search Platform, that is where the name Vespa comes from. More on the history in this blog post https://blog.vespa.ai/vespa-is-becoming-its-own-company/

>I don't see a ton of data that points towards us (vector DB ppl) building >towards traditional TREC/BEIR #s.

This is highly accurate, most vector database companies don't talk about the shortcomings of vector representations for search.

>TLDR: - I think pure vector search platforms should be evaluated differently than >traditional keyword search platforms - I think vector search is a tool -- simply >one of many -- that search engineers should use to make their search engine >results more relevant.

This is a contradiction. One one hand you say that you want to improve relevance, on the other hand you say that vector search as a tool cannot be evaluated as other models (tools).

We have plenty of open information retrieval datasets (both full retrieval and ranking) where you can compare different methods or tools and assess the relevance impact.

It's an opinionated blog post published on Arxiv, masquerading as research.

IMHO, it's a gigantic self-own and doesn’t promote Lucene in a good way. For example, by demonstrating how they get only 10 QPS out of a system with 1TB of memory and 96 v-cpu's (after 4 warmups).

The HNSW implementation in Lucene is fair, and within the same order of magnitude as others. But, to get comparable performance, you must merge all immutable segments to a single segment, which all Lucene oriented benchmark does, but which is not that realistic for many production workloads where docs are updated/added in near real-time.

Very good point, plus that if you introduce approximate search instead of exact search, you can only loose quality and in some cases the loss is so significant that it becomes worse than plain simple techniques.

Faiss implements multiple different ANN algorithms, and some like IVF can use disk based indexes.

Disclaimer, I'm a developer working on the Vespa.ai project. One reason is that we simply don't have DevRel teams or marketing teams, but we still have decent interest, from large companies like Spotify using Vespa in production for semantic search (at scale) https://engineering.atspotify.com/2022/03/introducing-natura...

On topic: IMHO a better title of the blog post would be:

Do you actually need to enable approximate vector search?

Which is an excellent question, because, introducing approximation also introduces accuracy degradation, plus that you need an ANN algorithm, which also has tradeoffs.

In Vespa, you can choose between exact and approximate, with related tradeoffs such as recall (accuracy), search speed, indexing throughput and resource footprint. Just start easy with exact search, then it's just change the schema definition of the tensor field to include `index` and Vespa will build the necessary data structures for enabling approximation.

That is a very good analogy!

Only Vespa allows you to index multiple vectors per schema field, avoiding duplicating all the meta data of the document into the "chunk", and avoids maintaining the document to chunk fan-out. See https://blog.vespa.ai/semantic-search-with-multi-vector-inde...

Yes, https://blog.vespa.ai/vespa-hybrid-billion-scale-vector-sear...

Friendly reminder, code that you can read is not open source. The particular repo does not have any license information and cannot be considered open source.

Interesting to see how deep learning is applied to search using learned vector representations of documents (podcasts) and user queries.

Great blog post, throughput is very often overlooked when talking about model inference.

>Why does 100M vectors not work in ES?

I think that it's important to mention Elasticsearch version in this context :=)

Full disclosure, I work on Vespa.ai, but I have a pretty good understanding of how NN and ANN works in ES.

Elasticsearch 7x only had support for brute-force exact nearest neighbor search (1) which you can surely run with 100M documents, you just need to have a lot of nodes to bring latency down to service latency SLA. Typically, a single threaded/single index segment query with 1M documents with 128 dims using float is 300ms with Elasticsearch 7x (3). Two shards/segments instead would bring it down to 150 ms and so on. But 100M becomes costly, but perfectly doable.

With Elasticsearch 8.0, there is support for doing approximate nearest neighbor search (2) using the HNSW support from Apache Lucene 9. This would bring latency down to low single digit ms (single threaded, single shard) with 1M documents. The biggest disappointment was probably that they don't integrate the approximate nearest neighbor search with regular query filters/terms. This will likely be coming in Elasticsearch 8.1 or so, once it's implemented in Apache Lucene 9.x.

(1) https://www.elastic.co/blog/text-similarity-search-with-vect... (2) https://www.elastic.co/blog/introducing-approximate-nearest-... (3) https://github.com/jobergum/dense-vector-ranking-performance

MS Marco is the largest relevancy collection in the open, see https://microsoft.github.io/msmarco/

Vector models are nothing but representation learning and applying the model out-of-domain usually gives worse results than plain old BM25. See https://arxiv.org/abs/2104.08663

A concrete example is DPR which is a state of the art dense retriever model for wikipedia for question answering, when applying that model on MS Marco passage ranking it performs worse than plain BM25.