News for hackers. Stuff that matters.
Re:Ask HN: Sorting massive text files? by meterplech on Saturday August 14, 2010 @12:53PM (#1603581)
If you have access to K (or you can use the free J), this is exactly the type of problem they can solve. If you put the file into a list x, ?x is the only command you need to get a list of uniques.

http://en.wikipedia.org/wiki/K_(programming_language)

Re:Ask HN: Sorting massive text files? by Groxx on Saturday August 14, 2010 @03:04PM (#1603826)
Does "put[ting] the file into a list" mean "loading the file into memory"?
Re:Ask HN: Sorting massive text files? by silentbicycle on Saturday August 14, 2010 @06:49PM (#1604195)
I'm pretty sure they mmap the file. J and K are both designed for doing an operation over massive collections. You'll have to read it off the disk one way or the other, though.