News for hackers. Stuff that matters.
Re:Remove first 300M lines from 700 GB file on 1TB disk by barbegal on Tuesday September 22, 2020 @09:58AM (#24554420)
The user wants to avoid having to rewrite the file to disk

"This is one option. Exporting the data from the old DB again, skipping the first 300 Mio entries and cp it again over to the new and import it there. It roughly takes a day. I just thought there must be a way to do it in place: the data are already there - just truncate the first part of it and continue. But I'm not in a desperate position. I have backups and no (extreme) time pressure. So in this sense it's an exercise."

Solutions such as: compressing, splitting into smaller files, using dd would require re-writing most or all of the file so are not what the user is looking for

Using fallocate to change where the file starts from would appear to be the correct solution but it would leave some unwanted data at the start of the file.

The best solution would be to use fallocate and then overwrite the unwanted lines in the first block with newline characters.

Re:Remove first 300M lines from 700 GB file on 1TB disk by drran on Tuesday September 22, 2020 @10:02AM (#24554459)
In gdb, conditionally break on fopen() call, then use fseek to skip first 300M lines, then continue import.
Re:Remove first 300M lines from 700 GB file on 1TB disk by aidenn0 on Tuesday September 22, 2020 @12:02PM (#24556126)
Someone else who actually knows how to use debuggers; a dying breed!
Re:Remove first 300M lines from 700 GB file on 1TB disk by Loranubi on Thursday September 24, 2020 @05:23AM (#24576815)
Don't forget about intercepting each ftell() though...
Re:Remove first 300M lines from 700 GB file on 1TB disk by dathanb82 on Tuesday September 22, 2020 @11:14AM (#24555424)
That’s both awful and awesome.