Printf debugging forever! ;-)

In my experience, the only way to produce safe multi threaded code is to isolate the threading parts and synchronization and stress test them early on. At this point I use a combination of prints, random delays and simple asserts on invariants that the code should hold. Most issues are reproduced within about 10 seconds of 100% CPU core utilization. Some nasty corner cases may require minutes of grinding away.

Multi threaded debugging is an unsolved problem. There are no universal solutions to this problem. Conventional tools often fail so badly that it is best to write and test your multi threading / synchronization code in isolation, and then applying it to the actual workload which is tested in a single thread.

Another option is trying to build a formally verified model ahead of time (e.g. using Spin - www.spinroot.com) and then write the actual program after the model has been verified.

If you have a race condition in a multi threaded program, a simple printf may be enough to change the timings and make the bug disappear. But a breakpoint and stepping in a debugger is guaranteed to completely change the timings (unless your timings are on the order of minutes).

Over the years, I had less and less use for intense debugging sessions, but I've been reading and writing logs all the time.

Today, I see server side code without logging statements, I immediately ask myself, how do the authors debug this thing?

Related:

> If you want more effective programmers, you will discover that they should not waste their time debugging - they should not introduce the bugs to start with.

That's basically what the quote is implying - you should not be throwing things together and hoping it will work after debugging, but carefully designing to avoid bugs, so there won't be much if any debugging required.

An analogy to this can be found in the aviation industry: it's known that pilots do make errors and planes do crash, but such errors are treated as unacceptable and they try their hardest to minimise them. It's been a very successful strategy.

It is great when these work. Or when you have a high level team that doesn't abuse the system at all and gets things done using these strategies. I have seen too many times where the result is a system that even the creator does not fully understand. Worse, what should be simple changes turn into giant modifications of a Rube Goldberg machine.

It's so much easier if you can search what has happened instead of guessing. Also customer experience is much better if you tell them that is fixed instead of asking for details :-).

Disclaimer: I work for log management company (Sumo Logic).