Re:How we use tennis rankings for fixing bugs
by theptip on Wednesday October 07, 2020 @04:55PM
(#24712610)
I've played this game before, but it ends up being really difficult to estimate the cost of a bug. E.g. what's the cost of not upgrading <libraryX> from v1.2.0 to v1.5.0? Or a theoretical security bug that you think is going to be quite hard to hit, but would be ruinous if it was exploited? Is that bug 0.001% or 0.01% likely to be triggered?
In practice it's impossible to get accurate ROI estimates for large chunks of the work you do. So being honest about that and using an approximation often doesn't cost you any appreciable accuracy -- and you will definitely spend less time trying to figure out how to precisely cost your bugs.
Don't get me wrong, I'm in favour of using ROI as a lens for prioritizing bugs, I've just never seen anyone make a credible estimate that's more accurate than the nearest order-of-magnitude.
I'm sure this is much easier the bigger you get; Google can estimate how many clicks are lost due to a bug, and clicks are of known value, so you can make a case for this bug being worth $Xm. But for startups I think this is unfeasible because you're working with such small sample sizes.
In practice it's impossible to get accurate ROI estimates for large chunks of the work you do. So being honest about that and using an approximation often doesn't cost you any appreciable accuracy -- and you will definitely spend less time trying to figure out how to precisely cost your bugs.
Don't get me wrong, I'm in favour of using ROI as a lens for prioritizing bugs, I've just never seen anyone make a credible estimate that's more accurate than the nearest order-of-magnitude.
I'm sure this is much easier the bigger you get; Google can estimate how many clicks are lost due to a bug, and clicks are of known value, so you can make a case for this bug being worth $Xm. But for startups I think this is unfeasible because you're working with such small sample sizes.