Re:Remove first 300M lines from 700 GB file on 1TB disk
by londons_explore on Tuesday September 22, 2020 @12:24PM
(#24556437)
Or, if it's JSON, just replace the first 300M lines with spaces.
You probably don't care if it's 299M rows, so just dd over the start of the file, guessing the number of bytes to write and choosing a low number to be sure. You'll know if you're missing records when you run a count(*) on the database when it's loaded.
Use some hex editor (most support files larger than ram) to check you have the necessary opening square brace at the start of the file, and to remove whatever partial record is there.
Sure, it isn't elegant, but is a solution most people can come up with without needing to spend 20 minutes hunting through man pages.
You probably don't care if it's 299M rows, so just dd over the start of the file, guessing the number of bytes to write and choosing a low number to be sure. You'll know if you're missing records when you run a count(*) on the database when it's loaded.
Use some hex editor (most support files larger than ram) to check you have the necessary opening square brace at the start of the file, and to remove whatever partial record is there.
Sure, it isn't elegant, but is a solution most people can come up with without needing to spend 20 minutes hunting through man pages.