Re:Remove first 300M lines from 700 GB file on 1TB disk
by arbitrage on Tuesday September 22, 2020 @09:19AM
(#24554062)
it opens up two file handles, reads from the first and writes to the second.
[ Parent
| on HN ]
Re:Remove first 300M lines from 700 GB file on 1TB disk
by tumetab1 on Tuesday September 22, 2020 @09:31AM
(#24554174)
That doesn't fully explain it but I think I got it, each line is overwritten with what it's place on the 300th line after.
Something like
readLineOffset = 301
writeLineOffset = 1
While not read eof
line = Read(readLineOffset)
Write(WriteLineOffset, line)
readLineOffset ++
writeLineOffset ++
Re:Remove first 300M lines from 700 GB file on 1TB disk
by ComodoHacker on Tuesday September 22, 2020 @09:53AM
(#24554371)
dd reads and writes in blocks of bytes, not lines. You have to figure out where 300th line starts beforehand.