News for hackers. Stuff that matters.
Re:Ask HN: Sorting massive text files? by brown9-2 on Saturday August 14, 2010 @12:02PM (#1603459)
The canonical Hadoop example is counting the number of words in a large text file http://wiki.apache.org/hadoop/WordCount

You could use the same approach to simply take the unique tokens of the output of the final "count".

May be overkill if you you can read the file in less than an hour, but this approach (divide and conquer) may be a good inspiration.

There are no comments on this comment yet.