News for hackers. Stuff that matters.
Re:Ask HN: Sorting massive text files? by pgbovine on Saturday August 14, 2010 @11:59AM (#1603446)
first i'd split it up into a bunch of smaller files, maybe 100 files that are each 350 MB?

then if you have multiple cores on your machine, you can run multiple instantiations of the same script on those smaller files and aggregate the results later.

if you're more ambitious, you could look into using a lightweight MapReduce framework

There are no comments on this comment yet.