News for hackers. Stuff that matters.
Re:Ask HN: Best “I brought down production” story? by pschoeps on Saturday June 26, 2021 @06:43PM (#27646683)
About 5 years ago when I was just starting out I found myself designing a responsive course builder. My solution to a responsive interface at this time involved sending a very large stringified HTML file over websockets.

This wasn't a huge problem, but the configuration on Action Cable (Rails wrapper around websockets), logged the entire contents of the message to STDOUT. At a moderate scale, this combined with a memory leak bug in Docker that crashed our application every time one of our staff members tried to perform a routine action on our web app. This action resulted in a single log line of > 64kb, which Docker was unable to handle.

All of this would have been more manageable if it hadn't first surfaced while I was taxiing on a flight from Detroit to San Francisco (I was the only full time engineer). I managed to restart the application via our hosting providers mobile web interface, and frantically instructed everyone to NOT TOUCH ANYTHING until I landed.

There are no comments on this comment yet.