News for hackers. Stuff that matters.
Re:Show HN: HyperLogLog intersections by stochastic_monk on Monday January 08, 2018 @02:22PM (#16099318)
Great work!

I haven't messed around with the count-min augmentation for HLLs, but I've had a lot of success in practice.

I've experimented with using HLL sketches for set operations for genome comparisons. The great thing about these operations is that the cost of operations scales with the size of the sketch, not the size of the dataset being sketched. Add SIMD acceleration, and you have an extremely fast, compact, accurate way to perform approximate set operations.

Unfortunately, the naive intersection operation (a sketch consisting of an element-wise 'min' of the counts in both sketches) does not perform well in cardinality estimation.

The recent Ertl paper [0] and associated code [1] puts a lot of effort into more accurate estimation and set operations. In my experiments, I've found that his modified estimation formula is always more accurate than the standard method and remains accurate to much higher cardinalities for a given sketch size.

My SIMD-accelerated, threadsafe HyperLogLog implementation in C++ (with python bindings) using this improved estimation method is available at [2]. Using this structure, I was able to perform over a billion full-genome Jaccard index calculations overnight using `bonsai dist` from [3].

[0]: https://arxiv.org/pdf/1702.01284.pdf

[1]: https://github.com/oertl/hyperloglog-sketch-estimation-paper

[2]: https://github.com/dnbaker/hll

[3]: https://github.com/dnbaker/bonsai

Re:Show HN: HyperLogLog intersections by xstartup on Monday January 08, 2018 @05:53PM (#16101613)
How do you find the relevant papers? Is there any tool for this task?
Re:Show HN: HyperLogLog intersections by stochastic_monk on Tuesday January 09, 2018 @05:27PM (#16110741)
To be honest, a combination of arxiv automailings, colleagues, and github. A lot of people I work with are interested in sketch data structures.
Re:Show HN: HyperLogLog intersections by dman on Monday January 08, 2018 @06:28PM (#16101928)
Google?