Re:Ask HN: Sorting massive text files?
by ralph on Saturday August 14, 2010 @02:16PM
(#1603739)
GNU sort has a --compress-program option that can help if you're really tight on scratch space, e.g. lzop(1). The -T option or TMPDIR environment variable lets you state a filesystem with more free space can be used instead.
[ Parent
| on HN ]
The -T option especially helps if you have more than one hard drive, and you're sorting a very large file where disk I/O is the bottleneck. You can use a second hard drive for the temporary files.