Re:Ask HN: Sorting massive text files?
by pjscott on Saturday August 14, 2010 @02:58PM
(#1603814)
It looks like GNU sort already does this internally, but with smarter temp file management. Increasing the memory size looks like a good idea, and if you have multiple hard drives, it may help to specify a temporary directory on a different hard drive with the -T option, since this problem looks IO-bound.
[ Parent
| on HN ]
I believe GNU sort is a single-threaded external multi-pass/multi-way mergesort.
http://en.wikipedia.org/wiki/External_sorting
The reason to split is not to replicate mergesort, but rather to get around single threading in GNU sort. Of course, there are better options if this is a regular task, but GNU sort just happens to be really common and easy if it's a one-off.
I do wish I had a better split (since split seems to read the whole file) and a better sort (that was parallel and reasonably common) though.