An overengineered solution to `sort | uniq -c` with 25x throughput (hist)

Re:An overengineered solution to `sort | uniq -c` with 25x t... by donatj on Monday October 27, 2025 @03:49AM (#45718351)

I created "unic" a number of years ago because I had need to get the unique lines from a giant file without losing the order they initially appeared. It achieves this using a Cuckoo Filter so it's pretty dang quick about it, faster than sorting a large file in memory for sure.

https://github.com/donatj/unic

[ Parent | on HN ]

Re:An overengineered solution to `sort | uniq -c` with 25x t... by kees99 on Monday October 27, 2025 @03:34PM (#45725321)

> As unic works with Cuckoo Filters, there is a very small probability a line will be wrongly marked duplicate. Lines will never be incorrectly marked as unique

Is this the right choice here? I can imagine quite a few situations where rare duplicate (false-positive unique) would be perfectly tolerable, while missing entry (false-positive duplicate) is a problem.

[ Parent | on HN ]

Re:An overengineered solution to `sort | uniq -c` with 25x t... by noamteyssier on Tuesday October 28, 2025 @12:09PM (#45734727)

Nice tool!

[ Parent | on HN ]

Re:An overengineered solution to `sort | uniq -c` with 25x t... by noamteyssier on Tuesday October 28, 2025 @02:31PM (#45736847)

I've actually added a benchmark for this specific task and added `unic` to it.

It may not be the most fair comparison because with these random fastqs I'm generating the vast majority of the input is unique so it could be overloading the cuckoo filter.

[ Parent | on HN ]