News for hackers. Stuff that matters.
Re:Remove first 300M lines from 700 GB file on 1TB disk by arbitrage on Tuesday September 22, 2020 @09:19AM (#24554062)
it opens up two file handles, reads from the first and writes to the second.
Re:Remove first 300M lines from 700 GB file on 1TB disk by tumetab1 on Tuesday September 22, 2020 @09:31AM (#24554174)
That doesn't fully explain it but I think I got it, each line is overwritten with what it's place on the 300th line after.

Something like

  readLineOffset = 301
  writeLineOffset = 1
  While not read eof
    line = Read(readLineOffset)
    Write(WriteLineOffset, line)
    readLineOffset ++
    writeLineOffset ++
Re:Remove first 300M lines from 700 GB file on 1TB disk by ComodoHacker on Tuesday September 22, 2020 @09:53AM (#24554371)
dd reads and writes in blocks of bytes, not lines. You have to figure out where 300th line starts beforehand.