Re:Remove first 300M lines from 700 GB file on 1TB disk
by ComodoHacker on Tuesday September 22, 2020 @09:57AM
(#24554416)
It's #1 answer. Nothing wrong except it isn't safe. Depending on type of crash you might not have a chance to see your printed offset.
[ Parent
| on HN ]
Re:Remove first 300M lines from 700 GB file on 1TB disk
by anon946 on Tuesday September 22, 2020 @10:33AM
(#24554812)
If you print the offset after a write has safely completed, then it should still work. The internal copy is idempotent, up to the number of bytes in the first 300M lines. In other words, assuming reasonable line lengths, you can safely re-do a copy operation. So if you crash, you may not have printed out the last offset, but you have recorded a recent offset. You can then just restart from that recent offset.
Re:Remove first 300M lines from 700 GB file on 1TB disk
by tzs on Tuesday September 22, 2020 @10:27AM
(#24554717)
The old fashioned way to deal with making sure you see your printed offsets is to actually print your offsets. You'd go to the machine room and run the command on the system console, which would typically be a hardcopy terminal like a DECwriter.
The trendy modern way, of course, would be to use the cloud. Upload the offsets to the cloud as they are produced, pausing until you get confirmation after each upload.
A more clever and/or foolhardy way is to put the crash recovery markers in the file itself. Place evenly spaced markers throughout the file, saving the data that they overwrite in a separate file. Then do your in-place file shifting. Finally, replace the markers with the saved data.