News for hackers. Stuff that matters.
Re:Remove first 300M lines from 700 GB file on 1TB disk by hamburglar on Tuesday September 22, 2020 @01:17PM (#24557230)
Oh, I think I see what you're doing here. Step 1 is actually an iteration over the entire file (I interpreted it as merely step 1 of a per-1k-block loop), effectively making a backwards (blockwise) copy of the original file but only using up 1kB of extra space at any given time. Step 2 is to truncate the "last" 300M lines, because those correspond to the first 300M lines of the original file. Step 3 is to re-reverse the temp file into the destination.

This is great, although the line-vs-block-orientation issues will be tricky to make sure you get right, as others have pointed out. The truncate will almost certainly need to leave a partial last block.

There are no comments on this comment yet.