News for hackers. Stuff that matters.
Re:Ask HN: Sorting massive text files? by Tichy on Saturday August 14, 2010 @12:00PM (#1603453)
Never did something like it, but can't resist thinking about it.

Maybe a kind of divide and conquer could work? Split into several files, do the sort | uniq on each of them. Then merge them, checking for duplicates on the way. I think merging should be almost as fast as line counting, at least linear in the size of the two files.

Edit: I guess it would be slower than counting, because presumably it would write to a new file (the merged file). But still, it should be linear.

There are no comments on this comment yet.