Re:Ask HN: Sorting massive text files?
by keefe on Saturday August 14, 2010 @04:48PM
(#1604006)
maybe you have to partition the file but I'm guessing random access will allow you to access it fine, I haven't ever poked a 35GB text file either.
first thing I'd try is probably pseudo-this:
while((entry=readDataPoint())!=null){
sha1e = sha1(entry);
//database has log N index on sha1e
if(database.contains(sha1e)) continue;
//database object figures out insert syntax
database.insert({sha1:sha1e,data:entry});
}
first thing I'd try is probably pseudo-this:
while((entry=readDataPoint())!=null){