News for hackers. Stuff that matters.
Re:Ask HN: Sorting massive text files? by naz on Saturday August 14, 2010 @12:01PM (#1603455)

   sort -u <filename>
would be faster than all that piping
Re:Ask HN: Sorting massive text files? by secretasiandan on Saturday August 14, 2010 @12:20PM (#1603514)
I've never used it but there's a -S flag for sort that allows you to specify buffer size. So maybe look into increasing that as large as you can.
Re:Ask HN: Sorting massive text files? by sqrt17 on Saturday August 14, 2010 @02:17PM (#1603743)
-S is quite useful. For bigger sorting tasks, I usually use -S50% to speed up the speed of sorting. (sort's default memory size must be about 1MB).

It may also help to use compression with the temporary files (something like "--compress-program lzop"), but I've never tried that.

Re:Ask HN: Sorting massive text files? by jemfinch on Saturday August 14, 2010 @12:59PM (#1603590)
Compared to the IO costs, the computational difference must be inconsequential.
Re:Ask HN: Sorting massive text files? by jey on Saturday August 14, 2010 @01:05PM (#1603604)
Right, but sort is apparently clever enough to do a disk-backed mergesort on a file (according to timr's comment), but it doesn't get a chance to recognize that you're sorting a file if you just pipe in the data.
Re:Ask HN: Sorting massive text files? by jemfinch on Saturday August 14, 2010 @08:58PM (#1604341)
Why did you get modded up? You're wrong. Sort will write temporary files to /tmp if you exceed its in-memory sort threshold.

Did you really think sort(1) just bails if you pipe it more data than can be contained in memory?

Re:Ask HN: Sorting massive text files? by ralph on Sunday August 15, 2010 @05:36AM (#1604859)
You're correct. Perhaps the up-voters were thinking it must just be better to report the piping. Which it is, but that doesn't make the comment valid.
Re:Ask HN: Sorting massive text files? by jey on Sunday August 15, 2010 @10:25AM (#1605176)
Er, you're right. I don't know what I was thinking.
Re:Ask HN: Sorting massive text files? by ori_b on Saturday August 14, 2010 @01:42PM (#1603690)
I believe it uses scratch space in /tmp to do it if you pipe it in. (The manpage says that it allows you to specify an alternative temporary directory than $TMPDIR, however)