During my postdoc, I wrote some code running parameter optimisation on a large dynamical system. It wasn't clever code, it was closer to a grid search than anything else. I spun it up across 200 cores of the local supercomputer, I waited 72 hours... and then nothing. I had forgotten to write the results to disk.