An overengineered solution to `sort | uniq -c` with 25x throughput (hist)

Re:An overengineered solution to `sort | uniq -c` with 25x t... by f311a on Monday October 27, 2025 @05:06AM (#45718757)

People often use sort | uniq when they don't want to load a bunch of lines into memory. That's why it's slow. It uses files and allocates very little memory by default. The pros? You can sort hundreds of gigabytes of data.

This Rust implementation uses hashmap, if you have a lot of unique values, you will need a lot of RAM.

[ Parent | on HN ]

Re:An overengineered solution to `sort | uniq -c` with 25x t... by noamteyssier on Tuesday October 28, 2025 @12:44PM (#45735212)

Yeah definitely, it's always a trade-off. I think in many cases where I use it especially the number of unique values is actually not crazy high (much less than the required RAM) and the number of lines is crazy high.

So in those settings I think it's absolutely worth it

[ Parent | on HN ]