Re:Remove first 300M lines from 700 GB file on 1TB disk
by zadokshi on Tuesday September 22, 2020 @10:32AM
(#24554802)
You just cat/pipe it to an Amazon s3 bucket (check out s3 command line tools), trimming the lines you don’t want, and piping through bzip2 as well if needed. Storage cost for an hour or so to hold in s3 it is quicker and cheaper than trying to muck around installing a second hard drive.
[ Parent
| on HN ]