Figure out the output size, and create a sparse file with that size. You'll be able to use it like a regular file, but blocks will only take up space once you write to them.
Now simply copy from the input to the output file, starting at the end. Truncate the input file whenever the disk fills up.
Definitely tune that. The file-system is probably working in 4KiB blocks so there would be no benefit in working on smaller ones. In fact as you'll write to each block four times you are probably wasting a lot of time and causing more wear than you need to.
I would suggest something far bigger (in the MiB range at least), especially on traditional drives where reducing head movements is key but random access latency on SSDs is not as close to zero as many people think particularly for small writes. Make sure that whatever you move the data with is loading that full block of your desired size into RAM before starting to write to the destination (i.e. perhaps pipe through something like "buffer -s 4M" if your read/write tool of choice does not support this directly).
In fact an SSD might internally be working with MiB sized blocks, another reason to work in that scale at least and be careful of alignment where you can detect it or the lack of it.
But yes, that solution would definitely work if carefully optimised.
> The correct solution, of course, is to buy a second disk.
Definitely. Must faster, and you still have the original data in-place so you can run checksums to verify the new copy before committing to use it.
Also if working with network attached storage you have an extra set of considerations that will change the optimal block size to work with.
I can pick up new technologies quickly and I am a good developer (at least my manager says so) but I can never come up with elegant solutions like this to what I call a whiteboard problem. This is the exact reason why I fail at those whiteboard interviews.
Any suggestions for me?
Picking up new technologies doesn't require creativity. Being a good developer (these days) also doesn't require much creativity, especially if you're just implementing spec and not designing systems.
The best way to get better at puzzles is to practice puzzles! The reason I don't want to call these problems whiteboard problems is because it implies that they're only used in that context. These types of puzzles can be done for fun, too! Perhaps you start with simpler puzzles and try some simple solutions. You might solve a few. You'll try some harder puzzles and you'll get frustrated because the solution doesn't come immediately. Sometimes you'll combine two solutions you've used previously to solve something more complex. It's an amazing feeling!
If you want a book of math puzzles, I recommend "Measurement" by Paul Lockhart. He explains and helps you explore math in the way he thinks it should be taught in schools, and I agree with him.
If you want something more programming focused, take a look at "Programming Pearls" by Jon Bentley. It's a very old book, so the constraints he has to work with are quaint by today's standards (do something without using more than a megabyte of RAM. Hah!), but they're good puzzles as far as I remember.
Good luck!
Edit to add: The most useful thing you'll get from spending time with puzzles from a certain field is the intuition about which tools to reach for and which to forget about. You'll still likely have to try for a bunch of tools before you find the ones that work the best (or at all), but you'll get better at choosing ones that are more likely to work sooner, so you'll be able to progress faster on harder puzzles. As Greg LeMond said about cycling: ‘[It] never gets any easier, you just go faster’ :)
I never had a single whiteboard interview in my life, so I can't give you advice on how to get better at them, sorry.
Back in university, I had a "Design and Analysis of Algorithms" course, and one of the first (toy) exercises was to implement a queue using other data structures. This solution with two stacks was what I came up with. That's 12 years ago now, good times :-)
Rehearsing your knowledge of data structures and algorithms while studying algorithmic quizzes can get you pretty far, it's time-consuming though so it needs to be something you get fun out of it, I don't get much fun out of it so I have done the bare minimum that is needed for this industry...
But the basic idea still works: You could read the original file forward to find the index of the 300M'th line, and then just halt the stack reversing bit when you get there. Delete the whole original file and then stack it back.
The sparse file suggestion a sibling made is really great. Otherwise, you could reverse the chunks as you write them, and end up with a fully reversed file, or just have to be careful about block boundaries and know that contents are forward, but blocks are backward. I wouldn't want to bulk read a 700g file with line processing, if it's possible to ignore that.
I think the only gotcha is to handle the last block read in step 1 in case it's not evenly 1 kiB. Otherwise the first read in step 3 will overlap the blocks.
This is great, although the line-vs-block-orientation issues will be tricky to make sure you get right, as others have pointed out. The truncate will almost certainly need to leave a partial last block.
Implementation sketch:
1) Successively read the last 1 kiB from the source file, append it to a temporary file, and truncate 1 kiB from the source file.
2) Truncate the last 300M lines from the temporary file.
3) Successively read the last 1 kiB from the temporary file, append it to the destination file, and truncate 1 kiB from the temporary file.
This way, you never need more than 1 kiB of additional disk space (tune this number as desired). I think this is an elegant solution. The correct solution, of course, is to buy a second disk.