Re:Ask HN: Sorting massive text files?
by pvg on Saturday August 14, 2010 @02:20PM
(#1603751)
Hence the split. It's still likely to beat the crap out of sorting and it might not be needed, depending on the data. I do wonder what the dataset is that makes for such tiny, short lines.
[ Parent
| on HN ]
Given the numbers, the need to dedupe, the poster's strong motivation and relative inexperience...
I'd guess it's a gigantic spam list.