News for hackers. Stuff that matters.
Re:Remove first 300M lines from 700 GB file on 1TB disk by hprotagonist on Tuesday September 22, 2020 @09:01AM (#24553912)
the only answer i’d use here is “go buy another 1TB drive and then ...”

$100 and a day and a half for shipping is a much better idea than playing russian roulette with work data by adjusting files in-place.

Re:Remove first 300M lines from 700 GB file on 1TB disk by nybble41 on Tuesday September 22, 2020 @02:41PM (#24558252)
> playing russian roulette with work data by adjusting files in-place

That can be avoided if your filesystem supports reflinks / copy_file_range (BTRFS, XFS, OCFS2). Just create a reflink of the file, drop all the blocks at the beginning of the reflink "copy" up to the one containing the line you want to start with using fallocate --collapse-range, and overwrite the remaining data up to the first line with some suitable fill character. For the JSON-encoded lines in this case spaces should work. Total disk space used: one data block for the padding plus some metadata for the reflink. No changes to the original file. No copying or relocating 700 GB of data. Quick, easy, and and safe. But you have to start with the right filesystem for it to work.

Re:Remove first 300M lines from 700 GB file on 1TB disk by Demiurge on Tuesday September 22, 2020 @10:46AM (#24555009)
Very recently I've been in a similar situation with a dedicated server. There was no way to add storage and even getting another server would have been pretty expensive. Luckily it was a raid, so I just had to double the storage temporarily :)
Re:Remove first 300M lines from 700 GB file on 1TB disk by speedgoose on Tuesday September 22, 2020 @11:20AM (#24555507)
I guess you weren't allowed to use a public cloud provider. Because you can always mount a network file system using sshfs, s3 like, or similar. Or upload the file on another server rented for one hour.
Re:Remove first 300M lines from 700 GB file on 1TB disk by Demiurge on Tuesday September 22, 2020 @11:52AM (#24555979)
I guess I could have asked for the CC information, setup a new AWS account, and have them pay for the storage and egress of 1.5TB of data. However, that very likely would have lasted quite a while as well, and in my case I had to minimize downtime as well. So, no, I would say even if you're allowed to do something, sometimes it is not the best solution.
Re:Remove first 300M lines from 700 GB file on 1TB disk by silon42 on Tuesday September 22, 2020 @09:21AM (#24554078)
I'd use the answer 2 (but attempt to just compress the output first, depending on type of data, leaving original file alone)
Re:Remove first 300M lines from 700 GB file on 1TB disk by abhayhegde on Tuesday September 22, 2020 @09:23AM (#24554097)
That is a nice approach, I agree. But from next time it is better to use the name of the answerer or link rather than saying 1st or 2nd answer since the position may change anytime depending on upvotes.
Re:Remove first 300M lines from 700 GB file on 1TB disk by _mkef on Tuesday September 22, 2020 @10:00AM (#24554436)
Agreed.

I can't imagine unless s/he wants to accidentally lose the data any better alternative.

Even if this works perfectly, you never want to delete data you can't easily retrieve again.

Next Monday the client might change their mind

Re:Remove first 300M lines from 700 GB file on 1TB disk by help56883 on Tuesday September 22, 2020 @09:16AM (#24554028)
Yeah that's right. Even 2 TB drives don't cost $100 though.
Re:Remove first 300M lines from 700 GB file on 1TB disk by hprotagonist on Tuesday September 22, 2020 @09:46AM (#24554318)
Samsung 850 EVO 250GB 2.5-Inch SATA III Internal SSD (MZ-75E250B/AM) https://www.amazon.com/dp/B00OAJ412U/ref=cm_sw_r_cp_api_fab_...

Ok, $110.

Re:Remove first 300M lines from 700 GB file on 1TB disk by hervature on Tuesday September 22, 2020 @10:00AM (#24554438)
Parent is agreeing with you. 2TB drives do not cost $100.

https://www.amazon.com/Seagate-BarraCuda-Internal-Drive-3-5-...

Re:Remove first 300M lines from 700 GB file on 1TB disk by hprotagonist on Tuesday September 22, 2020 @10:25AM (#24554670)
ah damn platters are cheap!

i forgot about not-SSD for a second there.

Re:Remove first 300M lines from 700 GB file on 1TB disk by zeroimpl on Wednesday September 23, 2020 @08:34AM (#24566245)
It’s not like you can’t test your script on another file first. If it works fine on a 1GB file with 300K lines, why wouldn’t it then work on the real file?

What would you do when the next time this happens it’s 300B lines from a 1PB file?

Re:Remove first 300M lines from 700 GB file on 1TB disk by fomine3 on Thursday September 24, 2020 @02:06AM (#24575748)
Backup is important even if you're confident.