I get anxious every time I have to debug live systems. Developers are lucky because we can reproduce and test fixes on non-live systems. As a bonus, we can step through the code at critical junctures and can examine the complete state of the system
I always liken it to being a detective trying to piece together a crime scene from incomplete clues. But now that I think about it, it's also quite a lot like trying to diagnose an illness from symptoms.
I had about 2-4 systems talking to each other for this project (that's counting the system that I wrote that works as a plugin/module loader and all of it's plugins as 1 system).
It's a relic from my past that I'm glad I left behind. Editing plugins live, writing everything in usually the one or two days of free time I had between school engagements, and a few other tricky things that happened, this was probably one of the most funny projects I worked on.
By the end of it, I had two config file formats for pretty much everything, I opted out of using the right tool for the job most cases, and I kept having to upgrade hardware which was a pain in the ass since I didn't have any setup instructions since I made the damned thing.
Following is a list of everything that's public.
https://github.com/gravypod/StarmadeWrapper
https://github.com/gravypod/StarmadeWrapper-API
https://github.com/gravypod/StarmadeDebugging
https://github.com/gravypod/Checkers
https://github.com/gravypod/Starmade-Log-Reader
https://github.com/gravypod/StarmadeWapper-Panel
https://github.com/gravypod/StarmadeWrapper-Claim
https://github.com/gravypod/StarmadeWrapper-Warps
https://github.com/gravypod/Starmade-Map
It was quite a bit of a mess and I can definetly agree, debugging live systems, debugging distributed systems, and debugging unverified/untested/undocumented systems is crap too.
I'm pretty sure that being a doctor is the same as this except thousands of times worse. The amount of times I've messed up when working on my spaghetti plate was more then I can count and I'm pretty sure the human body is more slopily put together then my leaning tower of crap.
We get to mess with large permutations of different causes and effects, that we stimulate, in a short time while that would be unethical for medical professionals.