News for hackers. Stuff that matters.
Re:Show HN: Command Line Tool to Sort CSV and TSV Files by M... by sigil on Saturday January 06, 2018 @01:50PM (#16086791)
For a dataset larger than physical memory, using a memory filesystem like tmpfs for the merge stage will either swap (|tmpfs| < |ram|) or deadlock (|tmpfs| >= |ram|).

Instead, your best bet in that case is to give sort as much physical memory as you can spare:

    sort -S 95% -k1 huge.tsv
Extra disk I/O is inevitable since your dataset doesn't fit in memory. At least during a merge sort your disk reads will be O(N) and sequentially ordered.

Note: in the special case that your dataset is slightly larger than physical memory, splitting it up in advance such that one of the `sort -m` input files lives on a tmpfs should indeed be faster.

Other things to check out if you need Very Fast Large Sorts:

- Use `sort --parallel=N` to use multiple cores. By default it only uses 1.

- Use `sort --batch-size=NMERGE` to increase the number of files merged at once. Otherwise you may be doing more mergesort stages than are necessary.

There are no comments on this comment yet.