Re:Remove first 300M lines from 700 GB file on 1TB disk
by ComodoHacker on Tuesday September 22, 2020 @09:57AM
(#24554416)
It's #1 answer. Nothing wrong except it isn't safe. Depending on type of crash you might not have a chance to see your printed offset.
Re:Remove first 300M lines from 700 GB file on 1TB disk
by anon946 on Tuesday September 22, 2020 @10:33AM
(#24554812)
If you print the offset after a write has safely completed, then it should still work. The internal copy is idempotent, up to the number of bytes in the first 300M lines. In other words, assuming reasonable line lengths, you can safely re-do a copy operation. So if you crash, you may not have printed out the last offset, but you have recorded a recent offset. You can then just restart from that recent offset.
Re:Remove first 300M lines from 700 GB file on 1TB disk
by tzs on Tuesday September 22, 2020 @10:27AM
(#24554717)
The old fashioned way to deal with making sure you see your printed offsets is to actually print your offsets. You'd go to the machine room and run the command on the system console, which would typically be a hardcopy terminal like a DECwriter.
The trendy modern way, of course, would be to use the cloud. Upload the offsets to the cloud as they are produced, pausing until you get confirmation after each upload.
A more clever and/or foolhardy way is to put the crash recovery markers in the file itself. Place evenly spaced markers throughout the file, saving the data that they overwrite in a separate file. Then do your in-place file shifting. Finally, replace the markers with the saved data.
Re:Remove first 300M lines from 700 GB file on 1TB disk
by emerged on Tuesday September 22, 2020 @10:02AM
(#24554460)
You can miss a printed offset and it still works. The file data exists in two places while writing the offset.
Re:Remove first 300M lines from 700 GB file on 1TB disk
by jrochkind1 on Tuesday September 22, 2020 @11:31AM
(#24555659)
That is basically the answer in the OP, do you suspect there's something wrong with it?
1) scan to find 300m line offset N 2) shift file from N to 0 using reads+writes 3) truncate last N bytes
You can print out the current offsets in case of a crash you need to resume from. You read exactly the size of the file once and write it exactly once, and it's safe.