Re:Show HN: Command Line Tool to Sort CSV and TSV Files by M...
by sigil on Saturday January 06, 2018 @01:50PM
(#16086791)
For a dataset larger than physical memory, using a memory filesystem like tmpfs for the merge stage will either swap (|tmpfs| < |ram|) or deadlock (|tmpfs| >= |ram|).
Instead, your best bet in that case is to give sort as much physical memory as you can spare:
sort -S 95% -k1 huge.tsv
Extra disk I/O is inevitable since your dataset doesn't fit in memory. At least during a merge sort your disk reads will be O(N) and sequentially ordered.Note: in the special case that your dataset is slightly larger than physical memory, splitting it up in advance such that one of the `sort -m` input files lives on a tmpfs should indeed be faster.
Other things to check out if you need Very Fast Large Sorts:
- Use `sort --parallel=N` to use multiple cores. By default it only uses 1.
- Use `sort --batch-size=NMERGE` to increase the number of files merged at once. Otherwise you may be doing more mergesort stages than are necessary.
To overcome the slowdown of disk I/O, perhaps a workaround could be to use mfs or tmpfs, maybe something like:
Personally, I gave up on sort for large files and use k/kdb+. I suspect it is faster for sorting than sort or the Go libraries, but I could be wrong.