A few months ago, I was poking around ptrace details for an incomplete (unstarted, really) side project, and I came across a mailing list message suggesting that debuggers should be using perf events instead of debug registers for doing hardware tracepoints.

A few days later (maybe literally the next day, I don't quite remember anymore), a coworker was having an issue where we needed tracepoints to catch who was overwriting bad memory, but the debugger was causing the thread race to not happen properly. So I looked up how to use perf to set hardware watchpoints, and we caught the issue almost immediately after that. I would not have thought to try that had I not been poking around with my side project.

Some of my key skills as a developer are my ability to debug issues quickly and my ability to spelunk through large code bases. These skills were developed largely through poking around code bases for incomplete side projects. Sure, the details of how CVS works is probably never going to come up, but when you stumble across time synchronization issues for code, or the different filesystem semantics of NFS--those details tend to be more relevant, and you might not think to look them up before embarking on the "useless" side projects.