Re:Ask HN: What's your biggest programming mistake?
by RiderOfGiraffes on Wednesday February 02, 2011 @11:47AM
(#2170891)
While testing a live update to a safety-critical traffic monitoring system I had four sessions running. Development, deployment, shadow, and a link to the live system.
I was checking the deployment to the shadow system, repeatedly running over the check-list to ensure that the deployment went smoothly, and then running a comparison between the shadow system and the live system to check that the update took place properly.
Between each test I took down the shadow system and brought it back up in a clean duplicate of the live system. Except on this one occasion when I had a brain-fart and killed the live system.
It all came back as designed within 30 seconds, re-initialised and restarted without loss of data, but it was a scary moment.
I was checking the deployment to the shadow system, repeatedly running over the check-list to ensure that the deployment went smoothly, and then running a comparison between the shadow system and the live system to check that the update took place properly.
Between each test I took down the shadow system and brought it back up in a clean duplicate of the live system. Except on this one occasion when I had a brain-fart and killed the live system.
It all came back as designed within 30 seconds, re-initialised and restarted without loss of data, but it was a scary moment.