News for hackers. Stuff that matters.
Re:Writing a document database from scratch in Go: Lucene-li... by jll29 on Tuesday March 29, 2022 @12:46AM (#30838872)
This implementation uses set semantics with "and" as the only Boolean operator (Boolean retrieval model) - the dominant paradigm of the 1960s library catalog search systems.

The Lucene library and engines on top (Elastic, Solr) use ranked retrieval, so resuls are scored and returned in order from most to least relevant. For example, the Vector Space Model (VSM) computes the cosine of the vector that represents the query (vector of all query term frequencies, other components 0) with the document vectors (vector of all document term frequencies, other components 0), and there are other models such as the Probabilistic Model (also known as BM25 or Okapi, after the version of the ranking formula used or the name of the software system implementing it, respectively) or DfR (Divergence from Randomness), another Information Retrieval (IR) model type.

Since 1972 terms are weighted with the popular TDIDF weighting scheme (Term Frequency times Inverse Document collection Frequency, Spärck Jones (1972) J. Doc.) and often apply length normalization. I'd argue that these are important ingredients in a minimal implementation (and actually the first Lucene version had them).

Without ranked retrieval there is no guarantee that the most relevant result in a list of 10,000 documents is not returned last.

Re:Writing a document database from scratch in Go: Lucene-li... by eatonphil on Tuesday March 29, 2022 @07:06AM (#30841045)
Good catch! But I don't think it matters because I didn't implement full text search so there is no question that all documents returned are equally relevant.