A journey to searching Have I Been Pwned database in 49μs

Re:A journey to searching Have I Been Pwned database in 49μs by rovr138 on Sunday March 01, 2020 @09:59PM (#22461258)

> P.S. if you need to sort a huge file, just literally use the UNIX/Linux `sort` command. No, it does not load it all into RAM. It knows how to do chunked sorts, dump temp files into /tmp, and then merge them. Old school UNIX tools are smarter than you think.

This so much.

I’ve worked with many devs and admins that don’t understand the tools that they have at their disposal on their systems. They end up trying to reinvent the wheel and their solutions usually don’t consider all the edge cases

[ Parent | on HN ]

Re:A journey to searching Have I Been Pwned database in 49μs by saagarjha on Sunday March 01, 2020 @11:09PM (#22461583)

Note that this does take a while; I let it go for about two hours before killing sort.

[ Parent | on HN ]

Re:A journey to searching Have I Been Pwned database in 49μs by rovr138 on Monday March 02, 2020 @03:04PM (#22468322)

     time gsort --parallel=2 -o $HOME/pwned-pass-sorted.txt pwned-passwords-sha1-ordered-by-count-v5.txt
         2890.06 real      1400.86 user       165.54 sys

48mins

`gsort` is GNU sort on coreutils (not the one included on macOS).

This is on a Mac Mini 2011 (5,1) with the 2.3GHz i5.

They really have a lot of things built into these tools :)

[ Parent | on HN ]