Relevance is really subjective, domain specific, requires intense amount of measurement and testing and many different ranking signals. Lucene is a toolbox for crafting many of these signals.
For example, he says
> However, as you can see, this vector space model does not explicitly require a higher ranking document to contain more query terms than a lower ranking one.
Well the way you get a higher similarity in a vector-space model is matching more terms. The caveat being that IDF and field length makes you also consider a term's specificity. So if you search for 'luke skywalker' you care more about the 'skywalker' match than the 'luke' match. But a match on BOTH 'luke skywalker' would score higher (field lengths being constant)
https://lucene.apache.org/core/3_5_0/scoring.html
"Lucene scoring is the heart of why we all love Lucene. It is blazingly fast and it hides almost all of the complexity from the user. In a nutshell, it works. At least, that is, until it doesn't work, or doesn't work as one would expect it to work. Then we are left digging into Lucene internals or asking for help on [email protected] to figure out why a document with five of our query terms scores lower than a different document with only one of the query terms."
See? Lucene people know about it but they just do not think it is a problem.
But I do.
T-Wand, though also uses vector space model, makes sure it is the case.
Those settings aside, arguably there are cases where fewer term matches could be more relevant.
For the search “Luke skywalker”
A tweet for example mentioning the term “Skywalker” once has much higher “Star Wars” aboutness than a move that uses skywalker one page and Luke pages apart.
That is the information density of Star Wars type content is far higher in the tweet, whereas eventually a book ends up using most English somewhere.
You had a wrong understanding. Now you are corrected. Let's move on.
Nobody said anything about "fewer term matches could not be more relevant". You are just making up straw men here. That's not the discussion we are having.
What I said is this, from a user point of view, it's not good to have a document containing fewer query terms to rank higher. This is a fact that even Lucene acknowledge (at least when they were version 3.5.0.). You have nothing to counter this fact.
There’s been extensive research justification behind the vector-space model. BM25 is the 25th iteration of a model and well tuned BM25 holds the highest non nueral performance on many tasks including question answering[1]. Research has long found including factors other than total term matches matters. Such as IDF[2] and field length[3].
Have you benchmarked your relevance assumptions similarly? If so I’d love to see them and learn more!
1 - https://www.elastic.co/blog/improving-search-relevance-with-...
2- https://www.researchgate.net/publication/238123710_Understan...
I was simply motivate my work, pointing out it's a problem that the current generation of search engine does not address.
What is "relevance", it is of course a context sensitive question.
You talk as if BM25 is the gold standard when it is not.
The research on this is all over the place. I just read an article that says that BM25 is way worse than alternative language models. You don't have to look far, for example this one that talks about Wand:
https://dl.acm.org/doi/10.1145/2537734.2537744
You know why I quit academia? It is useless arguments and virtual signalings like these.
I'd rather go out and build a damn thing that people like to use.
I merely pointed out that there's a problem, and I have a solution. I did not claim that my problem and my solution solve all problems. Isn't this obvious?
In my problem, my solution beats Lucene. It's as simple as that.
So if Lucene wants to be this infinitely configurable search library, it would be advisable to offer my solution as an option, or offer an better one that does something similar.
So far I have not seen any takers, only excuses.
"Better Relevance"
and I am simply saying, arguably, you can't make that claim without evidence.
(That said, I do find much to like about the article and the WAND / T-WAND explanation. I'm only pointing out you can't claim 'better relevance' without a benchmark to go with it.)
A good top-K algorithm should rank a document containing more user query terms higher than a document containing less number of user query terms. This makes perfect sense. Right?" This should have been changed in: "For the sake of this blog post I am interested in a simplified relevance concept: more query terms matched -> more relevance" In this way the blog would have not caused the outburst of (correct) indignation of the Information Retrieval community, that has been working for years on the topic, and knows very well how much complex is relevance.
Why should I care about what they think? In the end of the day, they are serving the users. They are either going to be left behind, or they have to change.
For example, the newer search engines, such us, https://github.com/meilisearch/MeiliSearch, also do what T-Wand does. Users are going to vote by their feet. The "information retrieval community" be damned. It's as simple as that. Got it?
And change they will. What "information retrieval community" are you talking about? Just a bunch of fresh graduates who just got their PH.D, and some college students. That's about it. I don't see any IR professors coming out defending them.
That's just wrong. Period. This is the reason I mention his job, because his job requires him to listen to the users, instead of taking something he learned in school and use that to talk condescendingly to his users. That's not gonna fly.
Given this context, I think whatever I said it is entirely appropriate.
I'm just trying to understand what the author meant and their intention for anyone that might revisit this thread later. I think what you're saying sounds plausible as an interpretation.
It is actually smug, to suggest the author, in this case, me, a computer scientist and a past professor who taught Information Retrieval class for more than 3 years, and who just came up with a new search algorithm, to "take a look at Wikipedia page for TF-IDF".
In case you have not read the article due to your smugness, my search algorithm also uses TF-IDF and vector space model.
Please do not dis-understand what is going on here: I am running a startup, and I am also old enough to not care about publications as much as people who are younger or in academia.
That's why I chose to reveal this in a blog post instead of hiding it until after my paper is published. Understood?
I also do know a lot about IR, because I taught Information Retrieval class for 3 years when I was teaching in university, I read research papers, and I just come up with a new search algorithm.
It's just that some people cannot accept that there are people who can cross fields with ease, make contributions quickly, and move on to the next field that pit their interests.
Yes, I am one of those people. In addition to HCI, I also published in the following areas: VR, DB, NLP, IR and Psychology. Sorry to hurt your feelings, but it is what it is. Accept it and move on.
I wrote a blog post for lay people to read. I am under no obligation to do an extensive literature review, I only need to mention the essentials.
You just cannot seem to swallow the fact that someone who does not have a Ph.D in IR or computational linguistics can come in to contribute quickly then move on to something else. It hurts your feelings deeply, does it? Heed my advice, just get over it, don't take your degree or your specialization too seriously. It does not matter.
What did I miss?
The PhD flex in jest by the blog post's author is a bit awkward but I don't know that I would characterize it as smug.
Any insight into why the author is perceived as smug would be appreciated.
I am obviously a Chinese.
In the eyes of racists, Chinese Americans are supposed to be timid and does not make any noises, but I do make noises, so I am perceived as a smug today because I said I am a Ph.D, something else next time I said something else.
It's not complicated.
OK, re-reading it, there are a few hints (a Chinese proverb, your nickname, etc.). But nothing "obvious".
Probably not the explanation we are looking for here.
It's good for you that you are not a racist (it's sad that such a normal thing has to be a compliment, but it is a compliment. Good for you.), but racists do look for these cues and make their judgement based on these.
It's an unfortunate current state of affairs that we have to live with. Ignoring it does not make it going away though. If you don't like it, you got to call it out whenever you see it, which I hope you do, instead of being taken back by it, as you seems to be.
Why do you have to use a throwaway account to say this, I am curious.
I could see calling it audacious. But, our industry advances on audacity.
> As someone who has a Ph.D. in Human-computer Interaction ;-), I feel like I am entitled to define a condition of "good" in relevance here. I hereby declare that:
>> A good top-K algorithm should rank a document containing more user query terms higher than a document containing less number of user query terms.
> This makes perfect sense. Right?
Also, “most search engines” don’t use vector space model as the only way to rank result, for example, page rank.
Edit: in some search scenarios finding the documents with the most query terms make sense, but Lucene can also rank using this metric. Still, applaud the author's effort in digging into research literature. Search relevance is very hard and standard off the shelf metrics like TF-IDF and page rank are often not enough. Good search usually requires deep understanding of the specific subject domain and hand-tuning tons of signals, many of which aren't even strictly based on search terms (e.g., previously purchased products on a store's website, geographic location, trending results).