News for hackers. Stuff that matters.
Re:Ask HN: Sorting massive text files? by lsb on Saturday August 14, 2010 @02:34PM (#1603783)
0) Get your data into Amazon's cloud, and rent a server on EC2 with 68GB of memory. Spot instances are a buck an hour.

1) Do sort -u --buffer-size=60GB FILE. You'll sort it all in memory, and that'll be a great speedup.

It's easier to scale up than scale out if you have the money, and your dataset fits in memory, so don't bother with Hadoop for something as simple as that. What do you want to do after you get the uniques?

Re:Ask HN: Sorting massive text files? by trin_ on Monday August 16, 2010 @12:38PM (#1608280)
but wouldnt that also include uploading 35gb of data and finally downloading xxGB (result size)? i think when you add this time you can just take your multicore desktop and let it run over night.