Re:Investigating the 900GB “Collection #1-5” password leaks ...
by munro on Friday March 08, 2019 @08:28PM
(#19343884)
I'm getting 2 minutes and 29.98 seconds, with my 2017 MacBook Pro with 8 cores + 16 GiB ram. I'm using GNU tools, could you share more details on how you're getting 12 seconds?
I didn't have your email30m dataset, so I fabricated one, mine is smaller at 522M and has less entropy, since 'm assuming everyone uses gmail & uses numbers for usernames. :)
I also shuffled the dataset, because that would be cheating.
I followed what you did, but it took way longer when running on my machine.
And just out of curiosity, it takes 1 minute 29 seconds to sort & uniq the presorted dataset.
[ Parent
| on HN ]
Re:Investigating the 900GB “Collection #1-5” password leaks ...
by mappu on Friday March 08, 2019 @10:26PM
(#19344253)
I ran all your commands above on Debian Buster (4-core i5 2500 from 2011, 8GB ram, SATA SSD). My /tmp apparently isn't a tmpfs mount.
Results: http://paste.debian.net/1072430/ It took 54s to sort|uniq and 38s for the presorted dataset.
EDIT: For the unsorted dataset: 35s for `sort --parallel=8 -u` and 37s for `sort --parallel=4 -u`.
Re:Investigating the 900GB “Collection #1-5” password leaks ...
by vthriller on Friday March 08, 2019 @09:05PM
(#19344010)
GNU sort generates a bunch of temporary files for large inputs, and for a lot of linux folks /tmp is mounted as tmpfs (i.e. it's RAM/swap-backed), but it might do something else on other platforms, or just locate temporary files on disk or something, so that might be one explanation.
Or it could be the difference between qsort() from different libcs.
Re:Investigating the 900GB “Collection #1-5” password leaks ...
by aw3c2 on Saturday March 09, 2019 @04:40AM
(#19345343)
http://0x0.st/zHb2.7z
All I did was place the file in a ramdisk.