I wonder how much of that is professional wisdom that embodies this lognormal distribution. Law isn't quite the same as dev, since there can be a lot of lost calendar time just due to certified mail, or court dates, or whatever. But I've found that when I apply this formula to my estimates, I'm a lot less anxious when I start to bang my head against unforeseen setbacks.
Of course, knowing which value to emphasize to a manager depends on why they want the estimate. Sometimes they really want the effort estimate, and sometimes they want the calendar estimate, and I'll sometimes give both if the context warrants it.
That and thinking that two 4h efforts can be completed in one 8h workday. MFer interrupt work so frequently how do they forget it happens? Forgot you had me in that BS 2h sales meeting "for show"!?
You.
Always give calendar estimates. That is, after all, what matters to coworkers, to the business, to partners, to the customer, and to users.
I'd suggest starting to keep track of how much of your time you can take to work on the thing you work on. If it's between 30 % and 90 %, include that factor in your estimation procedure.
Of course that's a sign of trouble, but it's natural for the stage this company is at. (Note: different company than the one that did the lognormal study I mentioned a few posts ago, and a different role for myself, much more directly exposed to customer issues).
In an environment like that, calendar estimates are completely nonsense because the priority list itself is volatile on the timescale of the estimate. But work estimates can still be stable; I can still accurately report that I have eight hours of work left on issue A, and when I provide that estimate along with work estimates on issues B and C which are also assigned as "top" priorities, management can use that to decide whether I'll finish issue this week or not.
If you're sufficiently informed to know what your priorities will be on the timescale of an estimate, you can make reliable calendar estimates.
Could you explain in what stage to switch between things so quickly few things are ever finished, and why it makes sense to estimate those things at all, in that context?
So if it's just you to a customer, 1 week becomes 2.
If it's you, reporting to your manager, to a customer 1 week becomes 4.
Although I generally don't apply multipliers this liberally, I do keep in mind how many stakeholders there are that can add uncertainty and friction to a project.
The presentation to the sales department was very different, and can be summarized as "we'll try to be clearer from now on to distinguish estimated delivery dates from committed delivery dates, and don't be surprised if they're months apart."
Software developers' expertise is in developing software, not in analyzing statistics to predict the future.
Developer estimates are one of the best data points you have to predict how long something will take, but if you think that you can just take them at face value, you are not doing your job as a manager. You need to compare them to past estimates and how those estimates panned out during development.
Which is a shame and causes so much wasted work. I wish we all got better at this.
I also think fitting the exact distribution is unimportant and uninteresting. The important thing is that you have roughly the right distribution and that it has the fat-tailed property of only slowly converging to normal.
Why the distribution doesn't matter beyond that is that, I think, finding the exact 95 % upper bound is probably impossible to do with any statistical significance, because of its rarity and how contributing factors change over time. Getting in the right ballpark matters a great deal.
Verifying that you get in the right ballpark (only 1 in 20 blow their committed date, and they do so independently in time, etc, the usual value-at-risk stuff) is fortunately also trivial, no matter the underlying distribution and it's parametrisation at the time.
I agree that it might not be terribly important practically, but I think it's interesting because it might hint something about the underlying nature of task planning. Sort of like how if you were measuring radioactive decay times and found a Poisson distribution, you might learn something about the underlying nature of radioactivity.
What it is exactly, I'm still not sure, but I do think there's something there to poke at.
I speculate this self-similar nature drives a lot of the other odd properties we observe in software development, but I haven't come up with a specific model yet.
I’d say that the CLT wouldn’t apply here since the sub tasks aren’t independent? E.g. a discovery when doing sub task C could mean a change to already-completed sub tasks A and B
What you're proposing is effectively a fancy "padding factor", it's like on Star Trek when Scotty says they can only go at Warp-8, but really, Warp-11 is possible.
Sometimes stakeholders do behave like toddlers and get red in the face and stomp around when a "promise" is broken because "we aren't there yet." (ok, maybe not literally, but whatever the adult version of a toddler flip-out is).
But most of the time people can understand that unexpected problems come up. It's perfectly fine to make log-normal plots and think about statistics, but everyone will much happier if you explain what's going on to the stake-holders instead of just giving "black-box" estimates using confidence intervals.
What the lognormal says is "it might even be done by late May, most likely by June, but almost certainly by December".
The dev department needs to allocate people, so for them, the most useful thing is the mean completion time. So for them, June is the important thing, although they want to be aware of potential delays in case they need to reshuffle.
The sales department is playing a very different game. Upset, impatient customers cost a lot, both in terms of sales and reputation, and can be very unforgiving. Sales much prefers to communicate a hugely pessimistic version of when something will be ready, that they have confidence in, and possibly be early, rather than a 50-50 estimate.
This is decision theory in action, where each actor combines the same probability distribution with different payoff matrices and so arrives at different conclusions suitable to their role.
To enable this, we clarified the language around "estimates", "commitments", and "promises", as shorthand for different intervals of the same underlying distribution.
The first one tells you effectively nothing. The second one you can build strategy on.
I agree that a statement like that would not be useful. What I am saying is that it's better to make a good effort at estimation but then keep the stake-holders informed about what is going on, about the challenges and the mid-project delays.
Any "95%" confidence calculation is kinda dubious. It would only work if you have meticulous and accurate records, have a long-enough history of delivering the same kind of product with the same team, the same customer-base, and the same tooling, oh yeah, and everyone has the psychological safety factor to be honest. If you work in a place where that exists, more power to you, but I think that's rare.
> What I am saying is that it's better to make a good effort at estimation
But what does that estimation mean? If we don't know after the fact whether the estimation was good or bad, then it's useless because it's an answer to an unknown question, and every stakeholder will interpret it differently. In order to know after the fact, we must know ahead of time whether we're aiming for 50 %, 90 %, or something else.
I doubt it. It's dicey to do estimations even when everything is well understood, but if you lack detailed and accurate records of past work for very similar projects... the best you can do is pad like crazy, cross your fingers, meet the deadline and say your confidence interval calculation was "accurate."
And of course it does depend on the nature of the contractual relationship with your client.
However, even if we charitably assume that the developers are accurately estimating the mean, that means that out of 100 projects, 50 of them will be delivered late. It is likely that you'd want to do better than that, and to do so in a principled manner, you're going to need to adjust the estimates that come out of the estimation process.
There are some possible exceptions. For example, some organizations may have a lot of work unplanned but that unplanned work is segregated to not affect people working on planned work, etc. For example, you may have operations and development teams where operation absorbs most of the unplanned stuff and only rarely bothers the development.
Now, just because work can be accurately planned doesn't mean the organization is efficient. As an example, military tends to be relatively quite good at planning, very inefficiently (ignore the latest failed high profile project...) On the other hand startups tend to be more efficient while also less predictable environments.
I like to think that at some scale of the project some level of trust in projections is necessary to be able to even organize the project. So some drop in efficiency is acceptable in order to be able to plan work.
Of course interruptions do incur a cost to context switch. But that probably won't explain how one bug fix that you thought would take an hour actually did, and another that you also thought would take an hour took five, especially if you did them both on the same day.
> The software department was under a lot of pressure to improve their forecasting, and were somewhat despairing that their estimates were off by a factor of about 1.6 on average
Hum, 1.6x average is actually extra interesting to me - if case time tracking is left on while people are in meetings/etc, this could translate to a perfect estimate with about 1/3 of their time spent on non-casework (meetings, co-worker interruptions, etc). That kinda sounds about accurate to me, depending on the day...
Interestingly, we did extensive time tracking on a multi-year in-house software project and collected data comparing the estimated completion time of tickets with their actual time.
The software department was under a lot of pressure to improve their forecasting, and were somewhat despairing that their estimates were off by a factor of about 1.6 on average, and sometimes a factor of 10 or more. This persisted in the face of all attempts to improve calibration. Managers were worrying that developers had no idea how long a task would take and estimation was futile.
When we plotted the data, in all cases, the actual time was very accurately fit by a lognormal whose scale parameter was precisely the predicted completion time. That is, whether the tickets were predicted to take 1 hour, 3 hours, 13 hours, or whatever, the histogram of their actual completion times followed the exact same shape but with a corresponding scale change on the x axis.
This told me that the developers actually have a really good understanding of the class of problem they're dealing with when they start a task. But sometimes tasks have multiplicative factors that make them take longer than you expect. Sometimes the bug turns out to be two bugs, and so on. Based on this analysis, I urged them not to consider it a prediction failure when a ticket takes 10 times longer than expected; that's just a property of the lognormal distribution, and that estimate likely did a good job of reflecting all available information at the time they made it.
Instead of changing the estimates, I suggested that we pick a safety factor for external facing commitments that reflects this distribution. Padding the estimate by 1.6 factor gives the mean, but if you want to make a commitment you can take to a customer, you can just extend the lognormal up to 95% confidence or 99% confidence or however trustworthy your promised commitment needs to be. Of course, a 99% confidence interval on a lognormal is a pretty big factor. But if that's better than running late, it is what it is.
Another interesting thing is that you'd expect, by the central limit theorem, that sufficiently large tasks would eventually become normally distributed rather than lognormal, because they're composed of a large number of subtasks. But it turns out that lognormals are a pretty pathological case; sums of n lognormals can continue to look nearly lognormal until n becomes really, really large.