Re:Remove first 300M lines from 700 GB file on 1TB disk
by drran on Tuesday September 22, 2020 @10:02AM
(#24554459)
In gdb, conditionally break on fopen() call, then use fseek to skip first 300M lines, then continue import.
Re:Remove first 300M lines from 700 GB file on 1TB disk
by aidenn0 on Tuesday September 22, 2020 @12:02PM
(#24556126)
Someone else who actually knows how to use debuggers; a dying breed!
"This is one option. Exporting the data from the old DB again, skipping the first 300 Mio entries and cp it again over to the new and import it there. It roughly takes a day. I just thought there must be a way to do it in place: the data are already there - just truncate the first part of it and continue. But I'm not in a desperate position. I have backups and no (extreme) time pressure. So in this sense it's an exercise."
Solutions such as: compressing, splitting into smaller files, using dd would require re-writing most or all of the file so are not what the user is looking for
Using fallocate to change where the file starts from would appear to be the correct solution but it would leave some unwanted data at the start of the file.
The best solution would be to use fallocate and then overwrite the unwanted lines in the first block with newline characters.