Sort will actually externally sort blocks into temp files and merge them. Adjusting this block size can help with thrashing.

Awk may still be better for uniquification.

What about 'sort -u' ?

I'm not sure. I haven't closely studied the difference between each algorithm. My guess would be that sort -u would perform better as the data set gets larger with a good block size setting because it does do an external sort. Cardinality would also affect the performance. If the unique set handily fits in memory, an external sort on a large data set wouldn't be very efficient.

Or you can use `sort -u` and not have to pipe to `uniq`.

Yes, awk does some real black bagic. It's awesome how it can parse really, really big files.