Re:Ask HN: Sorting massive text files?
by Tichy on Saturday August 14, 2010 @12:00PM
(#1603453)
Never did something like it, but can't resist thinking about it.
Maybe a kind of divide and conquer could work? Split into several files, do the sort | uniq on each of them. Then merge them, checking for duplicates on the way. I think merging should be almost as fast as line counting, at least linear in the size of the two files.
Edit: I guess it would be slower than counting, because presumably it would write to a new file (the merged file). But still, it should be linear.
Maybe a kind of divide and conquer could work? Split into several files, do the sort | uniq on each of them. Then merge them, checking for duplicates on the way. I think merging should be almost as fast as line counting, at least linear in the size of the two files.
Edit: I guess it would be slower than counting, because presumably it would write to a new file (the merged file). But still, it should be linear.