News for hackers. Stuff that matters.
Re:Why software projects take longer than you think: a stati... by jbay808 on Monday March 08, 2021 @06:54PM (#26393332)
> A reasonable model for the “blowup factor” (actual time divided by estimated time) would be something like a log-normal distribution.

Interestingly, we did extensive time tracking on a multi-year in-house software project and collected data comparing the estimated completion time of tickets with their actual time.

The software department was under a lot of pressure to improve their forecasting, and were somewhat despairing that their estimates were off by a factor of about 1.6 on average, and sometimes a factor of 10 or more. This persisted in the face of all attempts to improve calibration. Managers were worrying that developers had no idea how long a task would take and estimation was futile.

When we plotted the data, in all cases, the actual time was very accurately fit by a lognormal whose scale parameter was precisely the predicted completion time. That is, whether the tickets were predicted to take 1 hour, 3 hours, 13 hours, or whatever, the histogram of their actual completion times followed the exact same shape but with a corresponding scale change on the x axis.

This told me that the developers actually have a really good understanding of the class of problem they're dealing with when they start a task. But sometimes tasks have multiplicative factors that make them take longer than you expect. Sometimes the bug turns out to be two bugs, and so on. Based on this analysis, I urged them not to consider it a prediction failure when a ticket takes 10 times longer than expected; that's just a property of the lognormal distribution, and that estimate likely did a good job of reflecting all available information at the time they made it.

Instead of changing the estimates, I suggested that we pick a safety factor for external facing commitments that reflects this distribution. Padding the estimate by 1.6 factor gives the mean, but if you want to make a commitment you can take to a customer, you can just extend the lognormal up to 95% confidence or 99% confidence or however trustworthy your promised commitment needs to be. Of course, a 99% confidence interval on a lognormal is a pretty big factor. But if that's better than running late, it is what it is.

Another interesting thing is that you'd expect, by the central limit theorem, that sufficiently large tasks would eventually become normally distributed rather than lognormal, because they're composed of a large number of subtasks. But it turns out that lognormals are a pretty pathological case; sums of n lognormals can continue to look nearly lognormal until n becomes really, really large.

Re:Why software projects take longer than you think: a stati... by milesvp on Monday March 08, 2021 @07:39PM (#26393719)
I find this interesting. I've been told that law schools teach that when estimating how long something will take (not how many hours of effort, but calendar time) they say to double your estimate, then increase the units by one. So what you think is a 2 hour task will actually take 4 days to complete. 4 weeks will take 8 months.

I wonder how much of that is professional wisdom that embodies this lognormal distribution. Law isn't quite the same as dev, since there can be a lot of lost calendar time just due to certified mail, or court dates, or whatever. But I've found that when I apply this formula to my estimates, I'm a lot less anxious when I start to bang my head against unforeseen setbacks.

Of course, knowing which value to emphasize to a manager depends on why they want the estimate. Sometimes they really want the effort estimate, and sometimes they want the calendar estimate, and I'll sometimes give both if the context warrants it.

Re:Why software projects take longer than you think: a stati... by edoceo on Monday March 08, 2021 @08:48PM (#26394371)
This points to another issue I've observed is that managers/operators ask for an effort-estimate and then map that directly to calendar estimate.

That and thinking that two 4h efforts can be completed in one 8h workday. MFer interrupt work so frequently how do they forget it happens? Forgot you had me in that BS 2h sales meeting "for show"!?

Re:Why software projects take longer than you think: a stati... by kqr on Tuesday March 09, 2021 @01:01AM (#26395899)
Yeah, but who do you think are in the best position to know what your calendar will look like?

You.

Always give calendar estimates. That is, after all, what matters to coworkers, to the business, to partners, to the customer, and to users.

Re:Why software projects take longer than you think: a stati... by jbay808 on Tuesday March 09, 2021 @01:22AM (#26395995)
That depends on whether you're in a position to set priorities. I can tell you that something will take a week, but I can't tell you that it will be done next week unless I'm permitted to work exclusively on it this week.
Re:Why software projects take longer than you think: a stati... by kqr on Tuesday March 09, 2021 @02:09AM (#26396244)
Nobody is ever able to work exclusively on a thing for stretches of a week.

I'd suggest starting to keep track of how much of your time you can take to work on the thing you work on. If it's between 30 % and 90 %, include that factor in your estimation procedure.

Re:Why software projects take longer than you think: a stati... by jbay808 on Tuesday March 09, 2021 @10:29AM (#26399681)
I think we're talking past each other. I'm not referring to a percentage of time being consumed by meetings and such; I'm referring to how frequently I'm asked to multitask because several urgent projects are going on at once. New priorities are added unpredictability, and sometimes I'm asked to clear all other work off my plate to fight some fire.

Of course that's a sign of trouble, but it's natural for the stage this company is at. (Note: different company than the one that did the lognormal study I mentioned a few posts ago, and a different role for myself, much more directly exposed to customer issues).

In an environment like that, calendar estimates are completely nonsense because the priority list itself is volatile on the timescale of the estimate. But work estimates can still be stable; I can still accurately report that I have eight hours of work left on issue A, and when I provide that estimate along with work estimates on issues B and C which are also assigned as "top" priorities, management can use that to decide whether I'll finish issue this week or not.

If you're sufficiently informed to know what your priorities will be on the timescale of an estimate, you can make reliable calendar estimates.

Re:Why software projects take longer than you think: a stati... by kqr on Tuesday March 09, 2021 @11:08AM (#26400162)
Maybe I fail to see the point of even bothering to estimate something that is so unimportant it can be (or even appears likely to be?) down-prio'd in favour of more important things so quickly.

Could you explain in what stage to switch between things so quickly few things are ever finished, and why it makes sense to estimate those things at all, in that context?

Re:Why software projects take longer than you think: a stati... by edoceo on Tuesday March 09, 2021 @11:28AM (#26400391)
Early stage, need effort-budget to CBA vs risk to build calendar estimate.
Re:Why software projects take longer than you think: a stati... by ser0 on Monday March 08, 2021 @09:13PM (#26394605)
The rule of thumb that I was taught for software delivery is to double it for every layer of management you have to deal with.

So if it's just you to a customer, 1 week becomes 2.

If it's you, reporting to your manager, to a customer 1 week becomes 4.

Although I generally don't apply multipliers this liberally, I do keep in mind how many stakeholders there are that can add uncertainty and friction to a project.

Re:Why software projects take longer than you think: a stati... by chris11 on Tuesday March 09, 2021 @12:22AM (#26395682)
I've noticed that increasing the number of something usually indicates increases complexity. Working with another team takes more time than just keeping the work in one team. Working with two takes longer. Same with things like the number of repos, languages, or services.
Re:Why software projects take longer than you think: a stati... by kitd on Tuesday March 09, 2021 @04:03AM (#26396817)
That may be an expression of Brooks Law; increasing the number of people involved increases the average error count.
Re:Why software projects take longer than you think: a stati... by plutonorm on Monday March 08, 2021 @07:57PM (#26393882)
I bet that analysis went down like a lead balloon. Any other response than "we were a naughty department, but we have fixed the problem and are really sorry about it" would have fallen on deaf ears at all the places I have worked. Log-normal? Most people outside of tech dept dont know the difference between mean, median and mode.
Re:Why software projects take longer than you think: a stati... by jbay808 on Monday March 08, 2021 @09:09PM (#26394561)
Fortunately, this was a tech-internal meeting; management was sympathetic and pretty tech savvy and understood the takeaway, if not all the math. They had been legitimately worried that the department was not capable of holding to commitments even when they freely selected the date to commit to. And the developers themselves, to be fair, had no idea how good their estimates actually had been the whole time, because it had been viewed through the lens that a ticket that took 8 hours should have been estimated as 8 hours.

The presentation to the sales department was very different, and can be summarized as "we'll try to be clearer from now on to distinguish estimated delivery dates from committed delivery dates, and don't be surprised if they're months apart."

Re:Why software projects take longer than you think: a stati... by TheCoelacanth on Monday March 08, 2021 @09:27PM (#26394690)
I'm not sure if I would really consider that a legitimate worry.

Software developers' expertise is in developing software, not in analyzing statistics to predict the future.

Developer estimates are one of the best data points you have to predict how long something will take, but if you think that you can just take them at face value, you are not doing your job as a manager. You need to compare them to past estimates and how those estimates panned out during development.

Re:Why software projects take longer than you think: a stati... by jbay808 on Monday March 08, 2021 @09:44PM (#26394818)
Exactly, hence this endeavour! Where we discovered the secret map between the software developers' predictions and reality, which management worried was completely uncorrelated but it turned out to just be the lognormal distribution.
Re:Why software projects take longer than you think: a stati... by kqr on Tuesday March 09, 2021 @01:05AM (#26395913)
> Software developers' expertise is in developing software, not in analyzing statistics to predict the future.

Which is a shame and causes so much wasted work. I wish we all got better at this.

Re:Why software projects take longer than you think: a stati... by underdeserver on Tuesday March 09, 2021 @04:31AM (#26396975)
Yep. I would claim analyzing statistics to predict the future is an integral component of being a good software engineer.
Re:Why software projects take longer than you think: a stati... by kqr on Tuesday March 09, 2021 @12:53AM (#26395863)
I can independently attest all of your comment from my own internal research. Well summarised.

I also think fitting the exact distribution is unimportant and uninteresting. The important thing is that you have roughly the right distribution and that it has the fat-tailed property of only slowly converging to normal.

Why the distribution doesn't matter beyond that is that, I think, finding the exact 95 % upper bound is probably impossible to do with any statistical significance, because of its rarity and how contributing factors change over time. Getting in the right ballpark matters a great deal.

Verifying that you get in the right ballpark (only 1 in 20 blow their committed date, and they do so independently in time, etc, the usual value-at-risk stuff) is fortunately also trivial, no matter the underlying distribution and it's parametrisation at the time.

Re:Why software projects take longer than you think: a stati... by jbay808 on Tuesday March 09, 2021 @01:26AM (#26396020)
Thanks, and I'm glad to hear you've found the same thing! So including the article, that makes at least three of us, it seems.

I agree that it might not be terribly important practically, but I think it's interesting because it might hint something about the underlying nature of task planning. Sort of like how if you were measuring radioactive decay times and found a Poisson distribution, you might learn something about the underlying nature of radioactivity.

What it is exactly, I'm still not sure, but I do think there's something there to poke at.

Re:Why software projects take longer than you think: a stati... by kqr on Tuesday March 09, 2021 @02:14AM (#26396266)
One thing that makes software special and might contain a kernel of an explanation is that software is scale-free: big software is build from many small software, which in turn are build from even smaller software. This makes it different from e.g. houses, which are not built from many miniature houses.

I speculate this self-similar nature drives a lot of the other odd properties we observe in software development, but I haven't come up with a specific model yet.

Re:Why software projects take longer than you think: a stati... by ununoctium87 on Tuesday March 09, 2021 @05:19AM (#26397309)
> Another interesting thing is that you'd expect, by the central limit theorem, that sufficiently large tasks would eventually become normally distributed rather than lognormal, because they're composed of a large number of subtasks. But it turns out that lognormals are a pretty pathological case; sums of n lognormals can continue to look nearly lognormal until n becomes really, really large.

I’d say that the CLT wouldn’t apply here since the sub tasks aren’t independent? E.g. a discovery when doing sub task C could mean a change to already-completed sub tasks A and B

Re:Why software projects take longer than you think: a stati... by crispyambulance on Tuesday March 09, 2021 @06:27AM (#26397797)
> ...if you want to make a commitment you can take to a customer, you can just extend the lognormal up to...

What you're proposing is effectively a fancy "padding factor", it's like on Star Trek when Scotty says they can only go at Warp-8, but really, Warp-11 is possible.

Sometimes stakeholders do behave like toddlers and get red in the face and stomp around when a "promise" is broken because "we aren't there yet." (ok, maybe not literally, but whatever the adult version of a toddler flip-out is).

But most of the time people can understand that unexpected problems come up. It's perfectly fine to make log-normal plots and think about statistics, but everyone will much happier if you explain what's going on to the stake-holders instead of just giving "black-box" estimates using confidence intervals.

Re:Why software projects take longer than you think: a stati... by jbay808 on Tuesday March 09, 2021 @10:54AM (#26399997)
The idea here is just to be precise in what we're communicating, when providing a forecast, because different departments have different risk tolerances on that forecast.

What the lognormal says is "it might even be done by late May, most likely by June, but almost certainly by December".

The dev department needs to allocate people, so for them, the most useful thing is the mean completion time. So for them, June is the important thing, although they want to be aware of potential delays in case they need to reshuffle.

The sales department is playing a very different game. Upset, impatient customers cost a lot, both in terms of sales and reputation, and can be very unforgiving. Sales much prefers to communicate a hugely pessimistic version of when something will be ready, that they have confidence in, and possibly be early, rather than a 50-50 estimate.

This is decision theory in action, where each actor combines the same probability distribution with different payoff matrices and so arrives at different conclusions suitable to their role.

To enable this, we clarified the language around "estimates", "commitments", and "promises", as shorthand for different intervals of the same underlying distribution.

Re:Why software projects take longer than you think: a stati... by kqr on Tuesday March 09, 2021 @07:19AM (#26398112)
I disagree somewhat. There's a world of difference between saying "here's when we think it'll be done, but sometimes shit happens and there'll be a delay" and saying "we're 95 % confident it will be done by this date" -- especially when you have a solid track record behind the second.

The first one tells you effectively nothing. The second one you can build strategy on.

Re:Why software projects take longer than you think: a stati... by crispyambulance on Tuesday March 09, 2021 @12:41PM (#26401286)
> "here's when we think it'll be done, but sometimes shit happens and there'll be a delay"

I agree that a statement like that would not be useful. What I am saying is that it's better to make a good effort at estimation but then keep the stake-holders informed about what is going on, about the challenges and the mid-project delays.

Any "95%" confidence calculation is kinda dubious. It would only work if you have meticulous and accurate records, have a long-enough history of delivering the same kind of product with the same team, the same customer-base, and the same tooling, oh yeah, and everyone has the psychological safety factor to be honest. If you work in a place where that exists, more power to you, but I think that's rare.

Re:Why software projects take longer than you think: a stati... by kqr on Tuesday March 09, 2021 @11:44PM (#26407849)
I think I can work out a 95 % estimation I am willing to stake money on even without the things you mention. It does take good management/culture, but if I don't even have that, I'm headed for the dumpster anyway and should go home and rethink the whole thing first.

> What I am saying is that it's better to make a good effort at estimation

But what does that estimation mean? If we don't know after the fact whether the estimation was good or bad, then it's useless because it's an answer to an unknown question, and every stakeholder will interpret it differently. In order to know after the fact, we must know ahead of time whether we're aiming for 50 %, 90 %, or something else.

Re:Why software projects take longer than you think: a stati... by crispyambulance on Wednesday March 10, 2021 @07:50AM (#26410700)
> I think I can work out a 95 % estimation I am willing to stake money on even without the things you mention.

I doubt it. It's dicey to do estimations even when everything is well understood, but if you lack detailed and accurate records of past work for very similar projects... the best you can do is pad like crazy, cross your fingers, meet the deadline and say your confidence interval calculation was "accurate."

Re:Why software projects take longer than you think: a stati... by jbay808 on Friday March 12, 2021 @03:09PM (#26440306)
If the lognormal distribution is correct, the 95% confidence interval is a bit over 5 times the developer's gut-feeling estimate. The 99% confidence interval is 10 times the developer's gut-feel estimate. We found a lot of data to back this up, but your mileage may vary.
Re:Why software projects take longer than you think: a stati... by kybernetikos on Tuesday March 09, 2021 @08:54AM (#26398734)
This is a principled approach.

And of course it does depend on the nature of the contractual relationship with your client.

However, even if we charitably assume that the developers are accurately estimating the mean, that means that out of 100 projects, 50 of them will be delivered late. It is likely that you'd want to do better than that, and to do so in a principled manner, you're going to need to adjust the estimates that come out of the estimation process.

Re:Why software projects take longer than you think: a stati... by lmilcin on Monday March 08, 2021 @08:07PM (#26393996)
My understanding is that the "multiplicative factor" can typically be mostly attributed to what I call "quality of organization". Not to be confused with efficiency, the quality of organization is measure of how smoothly things go. For example, how likely it is that you are going to be distracted from your task, how much unpredictable synchronous coordination you need to complete your task, etc. An organization which can plan more of its work will have higher quality and an organization where more of its work is of unplanned nature will have lower quality.

There are some possible exceptions. For example, some organizations may have a lot of work unplanned but that unplanned work is segregated to not affect people working on planned work, etc. For example, you may have operations and development teams where operation absorbs most of the unplanned stuff and only rarely bothers the development.

Now, just because work can be accurately planned doesn't mean the organization is efficient. As an example, military tends to be relatively quite good at planning, very inefficiently (ignore the latest failed high profile project...) On the other hand startups tend to be more efficient while also less predictable environments.

I like to think that at some scale of the project some level of trust in projections is necessary to be able to even organize the project. So some drop in efficiency is acceptable in order to be able to plan work.

Re:Why software projects take longer than you think: a stati... by jbay808 on Monday March 08, 2021 @08:22PM (#26394147)
In this case, the time tracking was done in a very fine-grained way; you'd log your working time against a given ticket or other thing. So if you got interrupted to go to a meeting or something, that time got logged to the meeting, not the ticket.

Of course interruptions do incur a cost to context switch. But that probably won't explain how one bug fix that you thought would take an hour actually did, and another that you also thought would take an hour took five, especially if you did them both on the same day.

Re:Why software projects take longer than you think: a stati... by Izkata on Tuesday March 09, 2021 @11:01AM (#26400091)
> Interestingly, we did extensive time tracking on a multi-year in-house software project and collected data comparing the estimated completion time of tickets with their actual time.

> The software department was under a lot of pressure to improve their forecasting, and were somewhat despairing that their estimates were off by a factor of about 1.6 on average

Hum, 1.6x average is actually extra interesting to me - if case time tracking is left on while people are in meetings/etc, this could translate to a perfect estimate with about 1/3 of their time spent on non-casework (meetings, co-worker interruptions, etc). That kinda sounds about accurate to me, depending on the day...

Re:Why software projects take longer than you think: a stati... by jbay808 on Tuesday March 09, 2021 @11:31AM (#26400424)
This company handled time tracking quite differently than most, because everyone was paid by the hour, even devs. The same time tracking system was used to track time spent on tickets as was used to by the accounting department to track hours. If you went to a meeting you'd update your timesheet with the meeting and its billing code. People were quite religious about maintaining an accurate timesheet as part of the culture. I think it's about the highest quality dataset you could hope for.
Re:Why software projects take longer than you think: a stati... by jcims on Monday March 08, 2021 @07:31PM (#26393664)
This is pretty interesting, do you doing you are able to defend those estimates using the same methodology?
Re:Why software projects take longer than you think: a stati... by jcims on Tuesday March 09, 2021 @09:51AM (#26399308)
>do you doing you are

I have no idea