News for hackers. Stuff that matters.
Re:Ask HN: Sorting massive text files? by aristus on Saturday August 14, 2010 @01:52PM (#1603706)
The file has 2+ billion lines. A set or hash table won't work very if if there are, say 500 million unique strings.
Re:Ask HN: Sorting massive text files? by pvg on Saturday August 14, 2010 @02:20PM (#1603751)
Hence the split. It's still likely to beat the crap out of sorting and it might not be needed, depending on the data. I do wonder what the dataset is that makes for such tiny, short lines.
Re:Ask HN: Sorting massive text files? by aristus on Saturday August 14, 2010 @04:58PM (#1604024)
Given the numbers, the need to dedupe, the poster's strong motivation and relative inexperience...

I'd guess it's a gigantic spam list.

Re:Ask HN: Sorting massive text files? by pvg on Saturday August 14, 2010 @07:21PM (#1604237)
Hah, self-duh. Total failure of technical cynicism on my part.
Re:Ask HN: Sorting massive text files? by keefe on Saturday August 14, 2010 @06:46PM (#1604190)
an in-memory set won't work. indexes don't have to be in memory all the time to provide log N lookup, I mean just think about how b-trees work.

...and this is a problem most databases have addressed.

Re:Ask HN: Sorting massive text files? by pvg on Sunday August 15, 2010 @04:35AM (#1604815)
It might well work, it depends a great deal on the data. In any event, I was talking about in-memory hashing, the responder's assumption that I was was correct.