News for hackers. Stuff that matters.
Re:Parallel Locality Sensitive Hashing by paulgb on Friday May 30, 2014 @03:37PM (#7823560)
This is the first time I've heard of nilsimsa. Looking through the docs to find the algorithm, here's how it is described, but I'm still not clear on what a "character separation" is in this context or which 3 characters are chosen:

> Nilsimsa uses eight sets of character separations (character, character, etc.) and takes the three characters and performs a computation on them: (((tran[((a)+(n))&255]^tran[(b)]*((n)+(n)+1))+tran[(c)^tran[n]])&255), where a, b, and c are the characters, n is 0-7 indicating which separation, and tran is a permutation of [0-255]. The result is a byte, and nilsimsa throws all these bytes from all eight separations into one histogram and encodes the histogram.

I've found that in practice, shingling text and then minhashing it is scary-good at finding similar documents.

There are no comments on this comment yet.