Re:Learning from Your Bugs
by hankerfaster on Thursday April 28, 2016 @09:30AM
(#11588625)
We do something similar. Any bug that is submitted by a customer (rather than QA, or a trial team, or other developers) is analyzed. 50% of our bugs are due to poor requirements. 25% are due to poor design. The other 25% are just coding problems.
Most of the requirements problems derive from lack of understanding about the tradeoffs in the current system or lack of understanding what the customer actually wants.
Most of the design problems are a result of excessive complexity. People often forget to simplify and refactor prior to adding a new feature. They end up having to keep a larger system in their brain when designing than they otherwise need to. Therefore they miss interactions that exist or forget to handle them in the new design.
Most of the coding problems would have been caught had unit tests been clearer and closer to 100% branch/code coverage. It is amazing how many bugs lurk in the last 5% of testing.
It is actually quite fun to tell a product manager that they are the cause for a bug :) Usually it is the developer that
Re:Learning from Your Bugs
by chriswarbo on Thursday April 28, 2016 @09:55AM
(#11588842)
> Most of the coding problems would have been caught had unit tests been clearer and closer to 100% branch/code coverage. It is amazing how many bugs lurk in the last 5% of testing.
Presumably, coverage starts with the low-hanging fruit (simple, linear code; heavily-exercised code like startup routines and main loops; etc.) and grows to encompass ever more unweildy, smelly, edge-case parts of the codebase.
Hence, by increasing code coverage, we're boiling off the solvent of run-of-the-mill code, such that what remains uncovered gets an ever stronger concentration of hackery, unintelligibility and black magic. By 95% coverage, the bits remaining are those parts people don't want to touch, so I'd expect the concentration of bugs to be highest there.
Re:Learning from Your Bugs
by dfan on Thursday April 28, 2016 @08:16AM
(#11588207)
I do this with chess. I take positions from tournament games in which I made a correctable mistake (generally not positions in which there was just a lot of calculation to be done and I wasn't up to the task) and add them as flashcards to my spaced repetition system.
It is true that software engineers often treat bugs as unfortunate but unavoidable noise in the system; when one pops up, you fix it and move on. But it's really worth thinking them as an inherent part of the process of software development, one which you can improve and learn from, just like anything else.
Re:Learning from Your Bugs
by jondubois on Thursday April 28, 2016 @10:56AM
(#11589306)
I never bothered to write down details of bugs... I rarely learn a lesson from any one specific bug - Lessons are learned by being exposed to a wide range of different bugs over many years. Eventually it becomes gut instincts.
For example today I was debugging an issue with Kubernetes (it was a really cryptic error about client-server not being able to negotiate a protocol or similar). I had never used Kubernetes before and I couldn't find anything on Stack Overflow.
The first thing I did was check the client and server versions (to see if they matched) - They did so that wasn't the issue. So then I had a strong 'gut feeling' that it was related to the client configs.
In this case, my second hypothesis turned out to be correct. 5 years ago I would probably tested my way through 20 different hypotheses before finding the problem.
To summarize my debugging strategy, it's simple; just pick all the low-hanging fruits first. The hard part is knowing what a low-hanging fruit looks like.
Re:Learning from Your Bugs
by MattGrommes on Thursday April 28, 2016 @01:35PM
(#11590565)
This is basically intuition, a base of experiences that leads to these "gut instinct" decisions and feelings. Doing a journal like the author describes could actually help with your intuition because you're adding more specific information from each bug into your mental storage. Now you're getting a cursory level of data from each individual bug which leads you to think each one is less important. It's like skimming 3 books on a subject instead of close reading 1.
I'm just thinking out loud because I do what you describe here but I think the journaling idea could be valuable.
Re:Learning from Your Bugs
by vitd on Thursday April 28, 2016 @04:28PM
(#11591678)
FWIW, journaling has been useful to me. I've run into bugs and thought, "I know I hit something just like this before. Oh wait, let me look through my debug journal and see if there's something in there!" And often there is. It saves me having to do web searches or guess and use hunches. I can always fall back on those things if it's something I haven't seen before or is manifesting in a different way.
I'd go even further and say that I've often wasted time chasing down a hunch that was completely wrong.
Re:Learning from Your Bugs
by jondubois on Thursday April 28, 2016 @10:04PM
(#11593328)
This has happened to me several times as well (encountering a problem which I felt like I had encountered before on a specific project).
It's a good memory exercise to try to remember when was the last time I hit that bug. At the beginning, I wasn't very good (and I couldn't remember the solution of the earlier problem - a journal would have been handy at that point) but after doing this for a while, now I almost always remember the solution to various regressions I encounter.
I have a similar process myself. What I find to be helpful is to ask myself what, in particular, went wrong. If I have a handful of cases where "some api was used wrong", then I take that as a sign of a need for refactoring to make the callee easier to use.
Another that I like to ask myself, is how I can use the compiler and its type checking to my advantage. As in, how can I make it more difficult for myself to make this bug again in the future?
But I agree with the author's point: Nearly every bug is an opportunity to learn. You should not just fix bugs, but also to understand why they happened.
Re:Learning from Your Bugs
by gitmagic on Thursday April 28, 2016 @08:25AM
(#11588262)
I'm kind of doing the same but instead I write my analysis and notes in the pull request that fixes the bug. I think it's a great way for others to learn too :)
Most of the requirements problems derive from lack of understanding about the tradeoffs in the current system or lack of understanding what the customer actually wants.
Most of the design problems are a result of excessive complexity. People often forget to simplify and refactor prior to adding a new feature. They end up having to keep a larger system in their brain when designing than they otherwise need to. Therefore they miss interactions that exist or forget to handle them in the new design.
Most of the coding problems would have been caught had unit tests been clearer and closer to 100% branch/code coverage. It is amazing how many bugs lurk in the last 5% of testing.
It is actually quite fun to tell a product manager that they are the cause for a bug :) Usually it is the developer that
Presumably, coverage starts with the low-hanging fruit (simple, linear code; heavily-exercised code like startup routines and main loops; etc.) and grows to encompass ever more unweildy, smelly, edge-case parts of the codebase.
Hence, by increasing code coverage, we're boiling off the solvent of run-of-the-mill code, such that what remains uncovered gets an ever stronger concentration of hackery, unintelligibility and black magic. By 95% coverage, the bits remaining are those parts people don't want to touch, so I'd expect the concentration of bugs to be highest there.
:)
But really, at that point the business often doesn't care. Not when a requirement bug could cause months of lost engineer time.
Our experience is requirement bugs are much more costly than a missed unit test resulting in a bug.
It is true that software engineers often treat bugs as unfortunate but unavoidable noise in the system; when one pops up, you fix it and move on. But it's really worth thinking them as an inherent part of the process of software development, one which you can improve and learn from, just like anything else.
For example today I was debugging an issue with Kubernetes (it was a really cryptic error about client-server not being able to negotiate a protocol or similar). I had never used Kubernetes before and I couldn't find anything on Stack Overflow.
The first thing I did was check the client and server versions (to see if they matched) - They did so that wasn't the issue. So then I had a strong 'gut feeling' that it was related to the client configs.
In this case, my second hypothesis turned out to be correct. 5 years ago I would probably tested my way through 20 different hypotheses before finding the problem.
To summarize my debugging strategy, it's simple; just pick all the low-hanging fruits first. The hard part is knowing what a low-hanging fruit looks like.
I'm just thinking out loud because I do what you describe here but I think the journaling idea could be valuable.
I'd go even further and say that I've often wasted time chasing down a hunch that was completely wrong.
It's a good memory exercise to try to remember when was the last time I hit that bug. At the beginning, I wasn't very good (and I couldn't remember the solution of the earlier problem - a journal would have been handy at that point) but after doing this for a while, now I almost always remember the solution to various regressions I encounter.
Another that I like to ask myself, is how I can use the compiler and its type checking to my advantage. As in, how can I make it more difficult for myself to make this bug again in the future?
But I agree with the author's point: Nearly every bug is an opportunity to learn. You should not just fix bugs, but also to understand why they happened.
Will give it a go over the next while. Cheers for sharing!