News for hackers. Stuff that matters.
Re:Show HN: Didyougogo – An Altavista slayer by misterman0 on Sunday August 12, 2018 @11:58AM (#17744902)
I'm glad I could make you curious about this. I will gladly expand on the documentation around the language model and querying as soon as I can.

In broad terms: its a 16-bit vector space in which you can encode anything you like. I have chosen to encode phrases and words as bags-of-characters. This separates terms from each other enough that they can be searched for reliably (in almost all cases).

Terms that share a character have vectors that intersect one another and we can measure the cos angle between them. That's the score.

That is represented as a binary tree.

A scan in the tree gives you the closest match and an address into a file on disk; a list of document IDs.

At query time boolean logic is used on the result (document ID list) from each query clause (AND/OR/NOT key:value).

I'll write something up.

Re:Show HN: Didyougogo – An Altavista slayer by ryanfox on Sunday August 12, 2018 @12:13PM (#17744991)
Could the bags-of-characters approach cause issues with anagrams having the same vector?

It would be surprising (to me) to get the same results for e.g. "strange" and "garnets".

Re:Show HN: Didyougogo – An Altavista slayer by misterman0 on Sunday August 12, 2018 @01:53PM (#17745468)
Yes this model could cause issues such as the one you describe. With phrase queries/multi-token queries this becomes less of a problem. Phrases aren't anagrams that often.

A secondary index might become needed with the most popular terms, to resolve which anagram is the right one.