Re:Remove first 300M lines from 700 GB file on 1TB disk
by Liquid_Fire on Tuesday September 22, 2020 @11:46AM
(#24555897)
In that case, surely it would be possible to point the database importer at a pipe, the other end of which is the output of `tail -n +300000000 db_dump.json`. No need to actually modify the file.
Re:Remove first 300M lines from 700 GB file on 1TB disk
by londons_explore on Tuesday September 22, 2020 @12:24PM
(#24556437)
Or, if it's JSON, just replace the first 300M lines with spaces.
You probably don't care if it's 299M rows, so just dd over the start of the file, guessing the number of bytes to write and choosing a low number to be sure. You'll know if you're missing records when you run a count(*) on the database when it's loaded.
Use some hex editor (most support files larger than ram) to check you have the necessary opening square brace at the start of the file, and to remove whatever partial record is there.
Sure, it isn't elegant, but is a solution most people can come up with without needing to spend 20 minutes hunting through man pages.
If you read the comments on the StackExchange question, the scenario that prompted it does make sense:
>> I'm migrating our company's database and this is 700GB file with profile data in JSON format, each line one profile. The process that imported the data stopped after running several days after 70% done. Now, to not repeat the full import, I want to cut the first 70% (300 million lines).
>> [One option would be] exporting the data from the old DB again, skipping the first 300 Mio entries and cp it again over to the new and import it there. It roughly takes a day. I just thought there must be a way to do it in place: the data are already there - just truncate the first part of it and continue.
So it's not as if this is the only copy of the data. It would just be nice if there was a way to cut some of it without recopying everything.