News for hackers. Stuff that matters.
Re:Poor developer estimates, a.k.a. guesstimation by onion2k on Sunday September 15, 2013 @06:55AM (#6388698)
This is actually a solved problem in some industries using a technique called "3 point analysis". I've no idea why it hasn't really caught on among developers; it's very useful. Essentially it goes like this:

1. Estimate the very best possible outcome. How long would something take if everything went to plan?

2. Estimate the very worst possible outcome. If everything that can go wrong does, how long will the task take?

3. Estimate the most likely outcome. Knowing what you do about the client, the task, the tech, etc, how long is it likely to take.

(Point 3 is the normal way of estimating.)

The advantage of 3 point analysis is that you can use the difference between the times to estimate risk. If the best and worst outcomes are similar, then there's little risk of things going horribly wrong. If the best and the likely times are similar but the worst is much longer then there's something that can completely screw up - identify what that is and defend against it. If the likely and worst times are similar but the best time is much shorter, identify why that's the case and try to make sure it happens. And so on.

Better yet, by taking the aggregate best times for a set of requirements and comparing it against the worst, you can get a good view of the overall risk of a project.

Very useful indeed.

Re:Poor developer estimates, a.k.a. guesstimation by MetaCosm on Sunday September 15, 2013 @07:34AM (#6388771)
This is basically your solution: http://imgur.com/YNmKId9

PHB: Use the CRS database to size that market.

Dilbert: That data is wrong.

PHB: Then use the SIBS database.

Dilbert: That data is also wrong.

PHB: Can you average them?

Dilbert: Sure. I can multiply them too.

-----

I kid (sorta). It misses the core issue, why are developers estimates so terrible, 3 terrible estimates don't make a "good" one.

The post hits on some of the points, but IMHO misses a major one. Estimations are based on experience, yet developers are constantly building brand new types of products. If all you built was "blog engines" over and over again, you would get damned good at estimating them. The trick in software is as soon as something needs to be reused over and over again, it is abstracted, put in a library, built into a framework, so you can make your product different (read: build something entirely new) -- and EVERYONE sucks at estimating the cost of building things entirely new.

Think of the (entirely new in 1962) Lunar Module... they thought it would cost at max 350 million. It cost 2.2 billion. Those where very smart people with a major stake in getting the initial number right, but it was just a crazy ass guess because they had NEVER BUILT ONE BEFORE.

Re:Poor developer estimates, a.k.a. guesstimation by lobotryas on Sunday September 15, 2013 @08:37AM (#6388885)
You are absolutely right. It's impossible to estimate how long it will take to complete something you have never done before in your life. Unless the new code being written is absolutely trivial, estimates are going to be guesses, at best.

The only half-decent solution I found so far is iterative development. Take a feature and break it down i to big rocks. Further break down the big rocks into little chunks - each that you estimate would take you less than a day to complete.

Only after completing your first "big rock" in this way can you use your new data to gauge your velocity and give an estimate for completing the rest of the work that's worth the paper it's written on.

Re:Poor developer estimates, a.k.a. guesstimation by jacques_chester on Sunday September 15, 2013 @09:39AM (#6389046)
> Take a feature and break it down i to big rocks. Further break down the big rocks into little chunks - each that you estimate would take you less than a day to complete.

Experimental psychologists have found that decomposing tasks increases estimate accuracy, regardless of how or why the task is decomposed. It's called the "unpacking effect".

http://www.sciencedirect.com/science/article/pii/S0022103103...

Re:Poor developer estimates, a.k.a. guesstimation by webhat on Sunday September 15, 2013 @04:50PM (#6390282)
Unpacking works great when it's clear what the tasks are going to be. In my experience the bulk of the work does not go into the project or tool itself, i.e. completing the tasks, but rather into resolving issues.

I usually estimate that writing the code is anywhere from 10-20% of completing the project.

Some research shows that 1 kloc of code contains between 15 and 50 defects, and that code reviewing will pick up only 70-90%. This still leaves a lot of bugs. I sometime ago wrote up some meta research on estimating code reviewing: http://specialbrands.net/2011/10/24/code-review-in-scrum-xp-...

Re:Poor developer estimates, a.k.a. guesstimation by jacques_chester on Sunday September 15, 2013 @07:56PM (#6390834)
> I usually estimate that writing the code is anywhere from 10-20% of completing the project.

McConnell's book has a really excellent table of items that are left out of estimates. Nuts-and-bolts things like "API documentation", "deployment scripts", "status emails" and so on.

> This still leaves a lot of bugs.

Right. The classical software engineering theory is to build multiple quality gates and to study them for defects yielded. I recall that inspection stomps everything else; automatic testing came second.

The problem of time-to-fix is subtly different from time-to-release, though. Software with known defects is regularly in use.

Re:Poor developer estimates, a.k.a. guesstimation by kabdib on Sunday September 15, 2013 @01:33PM (#6389639)
Decomposing as far as you can is a good practice ... you'll still be way off, because you think you know all the problems that you're going to need to solve. So about halfway through the project your list is going to be longer and management is going to be freaking out.

The elephant in the room: Debugging. I don't know how many times I've been asked, "When are you going to find that bug?" Sometimes they are very easy. Sometimes they are very hard. For the really hard bugs, you need to be clever, scientific, determined, methodical and LUCKY. I don't know how to know when I'm going to run into some luck.

Re:Poor developer estimates, a.k.a. guesstimation by jacques_chester on Sunday September 15, 2013 @07:59PM (#6390850)
> you'll still be way off, because you think you know all the problems that you're going to need to solve

Absolutely; but I think it's reasonable to assume that a more accurate estimate is a more valuable estimate. You can also track estimate error over time and use that to adjust future estimates.

> The elephant in the room: Debugging. I don't know how many times I've been asked, "When are you going to find that bug?"

The problem is that in any sample of n=1, variability is huge. It's like saying "I am going to pick one person out of the global population. How tall is he or she?"

Well of course I can't know that. The sample is too small. But if instead I am told "we are going to pick ten thousand people at random from the global population; how tall are they in total?" then I can use previously-gathered statistics to give a range of estimates that will probably be accurate.

Re:Poor developer estimates, a.k.a. guesstimation by ExpiredLink on Sunday September 15, 2013 @01:00PM (#6389566)
> Estimations are based on experience, yet developers are constantly building brand new types of products. If all you built was "blog engines" over and over again, you would get damned good at estimating them. The trick in software is as soon as something needs to be reused over and over again, it is abstracted, put in a library, built into a framework, so you can make your product different (read: build something entirely new) -- and EVERYONE sucks at estimating the cost of building things entirely new.

That's it. Most, actually all, literature on software estimation I've seem ignores this simple fact. It's feasible to estimate cost and time for building houses, even skyscrapers, if you do that all the time. But how long does it take to climb a never-climbed mountain?

Re:Poor developer estimates, a.k.a. guesstimation by jacques_chester on Sunday September 15, 2013 @07:51PM (#6390818)
> But how long does it take to climb a never-climbed mountain?

This estimate could be derived in a number of ways. By analogy to other mountains, by creating a statistical model of climbing time based on parameters like gradient and climber skill, or by aggregating expert judgements of the mountain's likely climbing time.

A better example of a difficult-to-estimate task was the Manhattan Project. How long would it take to build the first atomic weapon? That was a truly novel problem, insofar as it would require fundamental physics research to solve.

But almost none of our work is of that nature. To be sure, there are unexpected requirements and complexities, but we are almost never performing fundamental research to achieve our ends.

Re:Poor developer estimates, a.k.a. guesstimation by jared314 on Sunday September 15, 2013 @10:10PM (#6391226)
> but we are almost never performing fundamental research to achieve our ends

"Fundamental research" is a matter of perspective. If a developer does not know Rails, he must perform fundamental research into building a new blog engine. While the unknowns, and variability, are significantly smaller, they are still unknown until he does the research. If she were to trod off the beaten path, even slightly, he could run into a community known, but developer unknown, roadblock/design flaw that didn't show up in basic research. (ex. A Rails blog engine vs a multi-threaded Rails blog engine)

Re:Poor developer estimates, a.k.a. guesstimation by jacques_chester on Sunday September 15, 2013 @11:08PM (#6391401)
While all estimates are relative to the context, I disagree that fundamental research is. Fundamental research is about discovering or inventing something nobody has ever done before.

What will be discovered or invented is an unknown; it may be unknown that it is unknown. Research is basically an open-ended search process. There's no foreseable end-state to base an estimate on -- though there are terminating conditions we can use to end research (ran out of money, made amazing discovery that will be passed to engineers to commercialise, gathered sufficient data to write publishable paper etc).

Take the linked post's example of AIDS research. How long will it take to cure AIDS? There is simply no way to know. We can produce estimates, but because it is a research problem, any such estimate would have to have wildly broad ranges to be accurate ("somewhere between 10 and 100 years").

In cases of pure research, you can't estimate the end-goal. Until it's over you may not even know that there were new end-goals you were unaware of (eg. 3M's Post It notes).

Re:Poor developer estimates, a.k.a. guesstimation by phillmv on Sunday September 15, 2013 @01:52PM (#6389684)
And even then, large capital projects are routinely wildly off estimates. I think the delta in "how far off we were" just tends to be larger more frequently.
Re:Poor developer estimates, a.k.a. guesstimation by jacques_chester on Sunday September 15, 2013 @08:01PM (#6390855)
In the Industrial Megaprojects book I quoted elsewhere, the single best predictor of every measure of project performance was "Front End Loading" -- how completely the project has been studied and specified before it is funded. There are at least three FEL stages. Companies that muck up FEL-1 are doomed to failure, companies that muck up FEL-2 or FEL-3 are doomed to poor returns.
Re:Poor developer estimates, a.k.a. guesstimation by jacques_chester on Sunday September 15, 2013 @09:20AM (#6388985)
You've gotten to the first step, which is to provide 3 estimates. The next step is to break estimates into smaller parts, estimate them individually, then roll it back up to build a pseudo-statistical profile of probable outcomes.

This is the PERT 3-point estimation technique. It's taught in every project management course in the world.

It so happens that almost nobody uses it in practice, because it is a pain in the rear end to do in Excel. It also so happens that I'm working on a tool to make it easy[1].

All I have right now is a landing page, proof-of-concept code and some articles. Oh, and a pile of research papers and books. It's turned out to be a deeper rabbit hole than I expected.

[1] http://confidest.com/

edit: how come "professional" is the only word that was spelled wrong on the landing page? Because Murphy is hilarious, that's why.

Re:Poor developer estimates, a.k.a. guesstimation by ams6110 on Sunday September 15, 2013 @12:36PM (#6389513)
It's turned out to be a deeper rabbit hole than I expected.

That's quite ironic, given the subject of this thread....

Re:Poor developer estimates, a.k.a. guesstimation by jacques_chester on Sunday September 15, 2013 @07:47PM (#6390809)
I keep a list of subjects for articles I want to write for the website.

One of them is "Why Research Is Almost Impossible To Estimate".

Re:Poor developer estimates, a.k.a. guesstimation by nradov on Sunday September 15, 2013 @12:09PM (#6389446)
Build your tool as a plug-in for existing popular project management tools like Rally. No one will use yours if they have to enter user stories and tasks in two places. I would love a tool that could analyze best / middle / worst case estimates for tasks plus account for predecessor relationships between user stories and generate a probability distribution chart of possible release dates.
Re:Poor developer estimates, a.k.a. guesstimation by jacques_chester on Sunday September 15, 2013 @07:36PM (#6390779)
I'm aiming to make it interoperable with other tools as a later step (first I need to launch the basic product). But it won't necessarily be a software project tool that's first off the rank.
Re:Poor developer estimates, a.k.a. guesstimation by chrisbennet on Monday September 16, 2013 @12:19PM (#6394001)
How long did you estimate this tool would take to write? :-)
Re:Poor developer estimates, a.k.a. guesstimation by dspeyer on Sunday September 15, 2013 @08:39AM (#6388891)
IIRC, 1 & 3 are usually the same, and 2 is usually still way too optimistic. And that's from research, not anecdote.

It still might be useful. Use 1 to anchor 3 to anchor 2 and then triple it.

See also: http://lesswrong.com/lw/jg/planning_fallacy/

Re:Poor developer estimates, a.k.a. guesstimation by ape4 on Sunday September 15, 2013 @10:43AM (#6389219)
What if the API you are planing on using can't do quite what is needed. But you only find out 2 weeks into the project. That's a worst case but its hard to foresee these things.
Re:Poor developer estimates, a.k.a. guesstimation by x0x0 on Sunday September 15, 2013 @05:06PM (#6390334)
that "solution" is, basically, ridiculous and not adopted for good reason.

A quote from Michael O'Church I quite like:

      Let's say that you have 20 tasks. Each involves rolling a 10-sided die. 
   If it's a 1 through 8, wait that number of minutes. If it's a 9, wait 15 
   minutes. If it's a 10, wait an hour.
      How long is this string of tasks going to take? Summing the median time 
   expectancy, we get a sum 110 minutes, because the median time for a task is 
   5.5 minutes. The actual expected time to completion is 222 minutes, with 5+ 
   hours not being unreasonable if one rolls a lot of 9's and 10's.
      This is an obvious example where summing the median expected time for the 
   tasks is ridiculous, but it's exactly what people do when they compute time 
   estimates, even though the reality on the field is that the time-cost 
   distribution has a lot more weight on the right. (That is, it's more common 
   for a 6-month project to take 8 months than 4. In statistics-wonk terms, the 
   distribution is log-normal.)
      Software estimates are generally computed (implicitly) by summing the 
   good-case (25th to 50th percentile) times-to-completion, assuming perfect 
   parallelism with no communication overhead, and with a tendency for 
   unexpected tasks, undocumented responsibilities, and bugs to be overlooked 
   outright.
   [1]
The point is, you are unable to know the worst possible outcome. Since it will have huge input into your average, your average is therefore useless. Perhaps the project isn't specced fully enough and, during implementation, a dev discovers an ugly or impossible feature interaction. You now have to either cut features or rearchitect another feature, ie the dice come up 10 in Michael's analogy.

An example from my background: I work on ml code. A learning algorithm was written in parallel with a particular loss function (essentially, this tells you if your estimates are good or bad). The person writing the spec assumed that a different loss function could be dropped in later. Indeed, this is how the math works on paper. But loss function number 2 required a different data layout between the machines. The vision was writing a new loss function should be one or two lines of code dropped into the middle of a hot loop; the reality was it took almost a month of work.

[1] https://news.ycombinator.com/item?id=3522910

Re:Poor developer estimates, a.k.a. guesstimation by jacques_chester on Sunday September 15, 2013 @07:38PM (#6390788)
> The point is, you are unable to know the worst possible outcome. Since it will have huge input into your average, your average is therefore useless.

Unbounded worst cases don't happen on every project.

Re:Poor developer estimates, a.k.a. guesstimation by jqjester on Monday September 16, 2013 @12:29AM (#6391629)
They don't have to be unbounded to be destructive to this estimation method, just large and unknowable.
Re:Poor developer estimates, a.k.a. guesstimation by jacques_chester on Monday September 16, 2013 @02:54AM (#6391978)
All estimates, in theory, have large and unknowable corner cases. Your problem may require the invention of a new algorithm. Your office may burn to the ground. An asteroid the size of Rhode Island could strike the planet. And so on.

All estimates are uncertain because there is no knowledge of the future. This is true of every profession, of every project.

When we make estimates we have to accept that they are uncertain and that the degree of uncertainty is governed partly by the problem domain.