News for hackers. Stuff that matters.
Re:Investigating the 900GB “Collection #1-5” password leaks ... by munro on Friday March 08, 2019 @08:28PM (#19343884)
I'm getting 2 minutes and 29.98 seconds, with my 2017 MacBook Pro with 8 cores + 16 GiB ram. I'm using GNU tools, could you share more details on how you're getting 12 seconds?

    $ time awk 'BEGIN{for(i=0; i<3100000; i+=1) for(y=0; y<10; y+= 1) print i "@gmail.com"}' > email30m.sorted
    awk  > email30m.sorted  8.29s user 0.82s system 97% cpu 9.352 total

    $ ls -lh email30m.sorted
    522M Mar  8 17:14 email30m
I didn't have your email30m dataset, so I fabricated one, mine is smaller at 522M and has less entropy, since 'm assuming everyone uses gmail & uses numbers for usernames. :)

    $ time shuf email30m.sorted > email30m
    shuf email30m.sorted > email30m  15.58s user 1.69s system 98% cpu 17.561 total
I also shuffled the dataset, because that would be cheating.

    $ wc -l email30m
    31000000 email30m

    $ time sort email30m | uniq > email30m.uniq
    sort email30m  873.55s user 4.29s system 585% cpu 2:29.98 total
    uniq > email30m.uniq  4.10s user 0.34s system 2% cpu 2:29.98 total

    $ wc -l email30m.uniq
    3100000 email30m.uniq
I followed what you did, but it took way longer when running on my machine.

    $ time sort email30m.sorted | uniq > email30m.sorted.uniq
    sort email30m.sorted  413.75s user 3.66s system 468% cpu 1:29.04 total
    uniq > email30m.sorted.uniq  4.11s user 0.40s system 5% cpu 1:29.04 total
And just out of curiosity, it takes 1 minute 29 seconds to sort & uniq the presorted dataset.
Re:Investigating the 900GB “Collection #1-5” password leaks ... by mappu on Friday March 08, 2019 @10:26PM (#19344253)
I ran all your commands above on Debian Buster (4-core i5 2500 from 2011, 8GB ram, SATA SSD). My /tmp apparently isn't a tmpfs mount.

Results: http://paste.debian.net/1072430/ It took 54s to sort|uniq and 38s for the presorted dataset.

EDIT: For the unsorted dataset: 35s for `sort --parallel=8 -u` and 37s for `sort --parallel=4 -u`.

Re:Investigating the 900GB “Collection #1-5” password leaks ... by vthriller on Friday March 08, 2019 @09:05PM (#19344010)
GNU sort generates a bunch of temporary files for large inputs, and for a lot of linux folks /tmp is mounted as tmpfs (i.e. it's RAM/swap-backed), but it might do something else on other platforms, or just locate temporary files on disk or something, so that might be one explanation.

Or it could be the difference between qsort() from different libcs.

Re:Investigating the 900GB “Collection #1-5” password leaks ... by aw3c2 on Saturday March 09, 2019 @04:40AM (#19345343)
http://0x0.st/zHb2.7z

All I did was place the file in a ramdisk.