News for hackers. Stuff that matters.
Re:Remove first 300M lines from 700 GB file on 1TB disk by jmt_ on Tuesday September 22, 2020 @08:54AM (#24553850)
The dd approach is interesting. Editing a file like that in place cowboy style with dd would fill me with anxiety however
Re:Remove first 300M lines from 700 GB file on 1TB disk by tumetab1 on Tuesday September 22, 2020 @09:18AM (#24554052)
Can anyone explain how dd is able to read and write a file at the same time without using a temporary file? :\
Re:Remove first 300M lines from 700 GB file on 1TB disk by tyingq on Tuesday September 22, 2020 @10:27AM (#24554709)
You can do it with a single filehandle, reading the data then seek()ing backwards before you do the writes. A Perl example that reads a file with the numbers 1 to 100 then replaces the digits with X's:

  #!/usr/bin/perl
  open(my $fh,">foo") or die($!);
  for (1..100) {print $fh "$_\n"};
  close($fh);
  open($fh,'+<','foo') or die($!);
  while(<$fh>) {
    my $spot=tell($fh)-length($_);
    seek($fh,$spot,SEEK_SET) or die($!);
    s/\d/X/g;#replace digits with X's
    print $fh "$_" or die($!);
  }
  close($fh);
If his file has lines of equal size, this would be fairly easy to do in place. Would be a little more complicated if not.
Re:Remove first 300M lines from 700 GB file on 1TB disk by yencabulator on Wednesday September 23, 2020 @07:50PM (#24573669)
Number of file handles is irrelevant, all handles for the one file are accessing the same data.
Re:Remove first 300M lines from 700 GB file on 1TB disk by tyingq on Thursday September 24, 2020 @01:54PM (#24581379)
Just pointing out it's not needed. Why is that irrelevant?
Re:Remove first 300M lines from 700 GB file on 1TB disk by seppel on Tuesday September 22, 2020 @10:28AM (#24554729)
There's nothing inherently wrong with reading and writing to the same file (as least if it is a binary file). And if you read at offset x and write at offset x-y you can easily move contents around with only a very small buffer. Just be sure that you move in the correct direction, so that you dont override stuff you need later.

That people normally use temporary files, has two reasons: The first is that it is much safer if something goes wrong. You then still have the original and not some half-way processed file. The other reason is that reading and writing in text mode does not really work. But you can of course process text files as binary files and deal with line endings yourself.

Re:Remove first 300M lines from 700 GB file on 1TB disk by arbitrage on Tuesday September 22, 2020 @09:19AM (#24554062)
it opens up two file handles, reads from the first and writes to the second.
Re:Remove first 300M lines from 700 GB file on 1TB disk by tumetab1 on Tuesday September 22, 2020 @09:31AM (#24554174)
That doesn't fully explain it but I think I got it, each line is overwritten with what it's place on the 300th line after.

Something like

  readLineOffset = 301
  writeLineOffset = 1
  While not read eof
    line = Read(readLineOffset)
    Write(WriteLineOffset, line)
    readLineOffset ++
    writeLineOffset ++
Re:Remove first 300M lines from 700 GB file on 1TB disk by ComodoHacker on Tuesday September 22, 2020 @09:53AM (#24554371)
dd reads and writes in blocks of bytes, not lines. You have to figure out where 300th line starts beforehand.
Re:Remove first 300M lines from 700 GB file on 1TB disk by netmonk on Tuesday September 22, 2020 @09:34AM (#24554199)
More over, how dd deal with EOL ? and counting them ?
Re:Remove first 300M lines from 700 GB file on 1TB disk by jrochkind1 on Tuesday September 22, 2020 @11:29AM (#24555641)
I believe it doesn't, and the solution in the OP first counts EOLs to get the byte offset you want, then tells dd what to do just in terms of byte offsets.