News for hackers. Stuff that matters.
Re:Ask HN: Sorting massive text files? by keefe on Saturday August 14, 2010 @04:48PM (#1604006)
maybe you have to partition the file but I'm guessing random access will allow you to access it fine, I haven't ever poked a 35GB text file either.

first thing I'd try is probably pseudo-this:

while((entry=readDataPoint())!=null){

       sha1e = sha1(entry); 

       //database has log N index on sha1e

       if(database.contains(sha1e)) continue;
 
       //database object figures out insert syntax

       database.insert({sha1:sha1e,data:entry});

}

There are no comments on this comment yet.