News for hackers. Stuff that matters.
Re:Remove first 300M lines from 700 GB file on 1TB disk by zadokshi on Tuesday September 22, 2020 @10:32AM (#24554802)
You just cat/pipe it to an Amazon s3 bucket (check out s3 command line tools), trimming the lines you don’t want, and piping through bzip2 as well if needed. Storage cost for an hour or so to hold in s3 it is quicker and cheaper than trying to muck around installing a second hard drive.
Re:Remove first 300M lines from 700 GB file on 1TB disk by saagarjha on Tuesday September 22, 2020 @12:37PM (#24556619)
You’ll need a good connection, though!