News for hackers. Stuff that matters.
Re:Remove first 300M lines from 700 GB file on 1TB disk by kingofpandora on Tuesday September 22, 2020 @10:35AM (#24554852)
What does "move everything else down" mean?
Re:Remove first 300M lines from 700 GB file on 1TB disk by kreetx on Wednesday September 23, 2020 @05:59AM (#24565091)
As someone not too familiar with precise system calls/file system internals, I just meant that if the "keep" part of the file starts at byte 1234, then start moving all of them at an above that location to locations at and above 0 (i.e to the beginning). And when the last byte is now at byte `last - 1234` then truncate the file.

One could write a tiny program to do that and I'd feel relatively safe running that.

(All that `dd` stuff would probably work too, and faster, but you'd need to spend time learning it first then. But that tool probably has discrepancies between OS'es, has history, etc. Can be simpler to just write the program.)

EDIT: and you can iterate in blocks of the size of the disregarded prefix.

Re:Remove first 300M lines from 700 GB file on 1TB disk by jgalar on Tuesday September 22, 2020 @11:36AM (#24555733)
I would guess memory map the whole file, then do the equivalent of:

memmove(beginning_of_file, beginning_of_file + 300_m_line_offset, file_size - 300_m_line_offset);