Have you tried adding whatever terms you want retrieved back as 'metadata' attached to your vectors?
This is highly accurate, most vector database companies don't talk about the shortcomings of vector representations for search.
>TLDR: - I think pure vector search platforms should be evaluated differently than >traditional keyword search platforms - I think vector search is a tool -- simply >one of many -- that search engineers should use to make their search engine >results more relevant.
This is a contradiction. One one hand you say that you want to improve relevance, on the other hand you say that vector search as a tool cannot be evaluated as other models (tools).
We have plenty of open information retrieval datasets (both full retrieval and ranking) where you can compare different methods or tools and assess the relevance impact.
I do see lots of us in the vector DB space demonstrating on real data the speed and semantic retrieval capabilities of vector DBs. Not many of us publish traditional IR benchmark #s simply because vector search is a pretty different ballgame than traditional keyword/BM25 search (as you know).
However, since most search practitioners have developed their mental models around things like TFIDF and BM25 + Learning To Rank, many vector DBs tailor their language to fit that heuristic.
If you look at vector DBs on their own -- I like to look at them as a piece of infra to use in conjunction with pre/post processing + RAG + BM25 + whatever else you want -- you'll find that measuring their accuracy against traditional IR benchmarks is a bit like fitting a square peg into a round hole. It almost fits! But not quite.
I think one of the starkest differences between the vector search and traditional search when it comes to retrieval evaluation is the great difficulty involved in creating judgment lists to measure vector search performance. Since vector search inherently retrieves documents that are approximately conceptually similar to a query, the initial recall set, especially with an IRL dataset (e.g. Reddit, Shopify, etc.) is a moving target. Language evolves. Idiosyncrasies pop up. Etc.
Not to say that those ^ issues don't plague keyword search as well. But the powerful thing about vector databases IMO is that unlike traditional indices that need lots of fine tuning and IR experts, a vector DB will capture these new additions to your domain's vocabulary immediately, based on the surrounding context in the document.
Of course you can do similar things with traditional search engines by incorporating curated lists of synonyms, antonyms, stop words, etc. but that takes time (despite engines like Solr having built-in lists of these kinds, search engineers often have to curate them further, manually, because their product's domain is so niche) and lots and lots of query-stream analysis (I remember personally doing lots of the latter at Reddit, hehe).
TLDR: - I think pure vector search platforms should be evaluated differently than traditional keyword search platforms - I think vector search is a tool -- simply one of many -- that search engineers should use to make their search engine results more relevant.
Disclaimer: I'm from Pinecone (and hi, Doug!)