Re:Remove first 300M lines from 700 GB file on 1TB disk
by OldHand2018 on Tuesday September 22, 2020 @10:37AM
(#24554889)
How many lines are in the 700GB file? What percentage of the file does the first 300M lines take up?
At work, we have lots of 20-30GB text files that gzip down to 14-15% of the original size. If this 700GB file was similarly populated, a gzip of the whole thing would be 100GB or so. Doing a tail piped to the gzip command would work.
At work, we have lots of 20-30GB text files that gzip down to 14-15% of the original size. If this 700GB file was similarly populated, a gzip of the whole thing would be 100GB or so. Doing a tail piped to the gzip command would work.