News for hackers. Stuff that matters.
Re:Remove first 300M lines from 700 GB file on 1TB disk by OldHand2018 on Tuesday September 22, 2020 @10:37AM (#24554889)
How many lines are in the 700GB file? What percentage of the file does the first 300M lines take up?

At work, we have lots of 20-30GB text files that gzip down to 14-15% of the original size. If this 700GB file was similarly populated, a gzip of the whole thing would be 100GB or so. Doing a tail piped to the gzip command would work.

There are no comments on this comment yet.