The Spark program can be applied as-is for data volume 10 times or 100 times bigger without being modified.

You would have to reinvent Spark if you started from Unix tools and applied it to bigger volume of data

One could of course discuss whether spark is the right tool for filtering a text file for a pattern, IMHO the post is interesting nevertheless.

    $ time awk 'BEGIN{for(i=0; i<3100000; i+=1) for(y=0; y<10; y+= 1) print i "@gmail.com"}' > email30m.sorted
    awk  > email30m.sorted  8.29s user 0.82s system 97% cpu 9.352 total

    $ ls -lh email30m.sorted
    522M Mar  8 17:14 email30m

    $ time shuf email30m.sorted > email30m
    shuf email30m.sorted > email30m  15.58s user 1.69s system 98% cpu 17.561 total

    $ wc -l email30m
    31000000 email30m

    $ time sort email30m | uniq > email30m.uniq
    sort email30m  873.55s user 4.29s system 585% cpu 2:29.98 total
    uniq > email30m.uniq  4.10s user 0.34s system 2% cpu 2:29.98 total

    $ wc -l email30m.uniq
    3100000 email30m.uniq

    $ time sort email30m.sorted | uniq > email30m.sorted.uniq
    sort email30m.sorted  413.75s user 3.66s system 468% cpu 1:29.04 total
    uniq > email30m.sorted.uniq  4.11s user 0.40s system 5% cpu 1:29.04 total

Results: http://paste.debian.net/1072430/ It took 54s to sort|uniq and 38s for the presorted dataset.

EDIT: For the unsorted dataset: 35s for `sort --parallel=8 -u` and 37s for `sort --parallel=4 -u`.

Or it could be the difference between qsort() from different libcs.

All I did was place the file in a ramdisk.

GNU sort can already utilize multiple cores, and you can override internal heuristics about whether it's going to be beneficial for one particular input or not with --parallel=N. Anyways, if multithreaded sort is available what you should probably do is replace `sort | uniq` with just `sort -u` to save some time on pipe IO.

`sort -u` is just as fast as `sort | uniq` for me but good call :)