Re:Ask HN: Sorting massive text files?
by pbh on Saturday August 14, 2010 @01:09PM
(#1603615)
Personally, I would run:
You should be aware that you may not get what you expect from sort unless you set LC_ALL="C". You should also pass "-S" to sort to set the main memory size, probably to a value like "1G". Of course, at some point when you're doing a lot of distributed processing you'll just want to use Hadoop, as other posters have noted.
[ Parent
| on HN ]
It looks like GNU sort already does this internally, but with smarter temp file management. Increasing the memory size looks like a good idea, and if you have multiple hard drives, it may help to specify a temporary directory on a different hard drive with the -T option, since this problem looks IO-bound.
I believe GNU sort is a single-threaded external multi-pass/multi-way mergesort.
http://en.wikipedia.org/wiki/External_sorting
The reason to split is not to replicate mergesort, but rather to get around single threading in GNU sort. Of course, there are better options if this is a regular task, but GNU sort just happens to be really common and easy if it's a one-off.
I do wish I had a better split (since split seems to read the whole file) and a better sort (that was parallel and reasonably common) though.