If you estimate in t-shirt sizes people will come to false agreements. The only thing that actually stands up to hindsight verification is estimating in as concrete units as possible – ideally calendar dates.
Of course, if you invoke statistical properties of past estimations to convert t-shirt sizes to calendar dates, you're sort of doing this. But why do it behind people's backs? In my experience (and drawing from system theory) it is much more useful to allow people to receive the feedback of their successes and failures.
There are only 2 reasons to do the translation behind peoples' backs AFAIK:
1) also from kahneman et al we know that humans are bad at estimating time, and much more consistent and accurate about measuring effort. And consistency is required for the law of large numbers to apply. By estimating in consistent non time units you avoid those cognitive biases.
2) humans will very quickly start to unconsciously bias the units if they feel pressure. Speaking in terms of time puts one very close to a lot of pressure points for a lot of people.
Ideally engineers are concerned with doing things Right, and PMs are using past performance to predict how long that will take.
Excuse the snark, but a lot of business decisions are made based on how much Project X will cost. You likely get paid based on time, so it's quite natural to gravitate toward time units - any intermediary unit is just an abstraction.
And how would you even say what a project will cost? Projects change all the time, and there is no way to know the entire scope beforehand unless it's really really small. Otherwise you're just lying.
Genuinely how does this work?
Or if you're getting building work on your house, they'll caveat that it'll cost you £X, saving any unforeseen circumstances that arise (if you've asbestos in your attic, they're not going to stick to the original price and just eat the cost)
In consulting, most clients want a fixed cost for a fixed scope, with any deviations coming as change requests & billed accordingly. Generally people are hestitant to agree to an agile model unless you've already got a lot of trust built up.
The vendor ends up losing money and the client gets a deal on paper but not in reality when you realize the corners that were cut to shove a product at the door.
You could generally go back and look at the emails on the project and see the exact point in time (often about 2/3 in) were the vendor realized they were losing money and started cutting whatever possible to not lose their shirt on the project.
I suppose that if projects were relatively straightforward like “we’re building you a wordpress site with X number of templates” one may have a chance at estimating upfront- but I’ve never worked on projects that predictability.
(of course changes are never free, but we all seem to want to brush that away, to the detriment of the customer.)
there are a lot of software projects out there where the customer isn’t chasing some shiny new toy. the folks who wrote the firmware for your microwave had a really good idea what the entire feature set was going to be before they started.
A sophisticated enough client can design a fixed-price deal that can completely devastate a software development firm if the guardrails are not in place. On the other hand, a sophisticated dev firm can design a fixed-price deal that will nickle and dime a client in perpetuity if the client doesn't recognize the trap.
Fixed-price is like handling dynamite, you have to know what you're doing.
And yes it is an abstraction, which is the whole point.
Human brains are extremely bad and inconsistent at estimating time, and much better at estimating effort. So you abstract and let them estimate in the terms they're better at. Not to mention, the abstraction means you average in unpredictable disruptions like illness, windows updates, and other events. Abstracted unit estimates are demonstrated to be much more accurate than (not-abstracted) time estimates.
I hate pretending that we're estimating 'complexity'.
It's not pretense. WTF should an engineer care about time? An engineer has to care about Done Right. It takes as long as it takes. A PM cares about time, and they can looks at the past history of similar tasks to estimate it very accurately. The abstracted "effort" unit is a way to identify "similar tasks".
It's also very well demonstrated that human brains suck at estimating time. Estimating difficulty is way more consistent.
But if you hate thinking in terms of complexity you can do the same thing by making sure your tasks are evenly sized, and averaging task throughput over time. Or if you also hate your pm you could tag similar tasks and have them do the leg work without the math to help.
In other words 'what is the average conversion from complexity to time'; in other words 'let us try to point time but call it complexity'.
> WTF should an engineer care about time? An engineer has to care about Done Right. It takes as long as it takes.
That's great, but then #points in a fixed-time sprint should be expected to be all over the place.
Your epic and VeryNecessaryFeature(c) takes 30 gold stars.
Pointy hair boss: this takes 3 weeks. (and make it 2)
How is this different from skipping the proxy and just using time directly?
No non-unit-of-time estimation system will matter if there isn't buy-in from all levels. If the Directors/VPs/C-Suite etc view development in assembly line terms, then Xpoints = Ytime can never be avoided and yes, it just becomes an exercise in frustration.
To date I haven't found a good antidote other than avoiding workplaces that view estimations as exactimations of time.
And that number might change over time and that’s okay, and you can keep tracking it.
It’s a bit of a mind trick.
Instead of thinking: “How long will this ticket take?” you think: “Is this rather as complex as the 3 star tickets I did in the past, or is it as complex as the 5 star tickets? “
You estimate in terms of other tickets, not in terms of time.
It helps to remove bias.
Nothing you said changes the main problem: gold stars are weeks and we all know it. Next year they are two weeks, fine.
"No I said it was 12 jellybeans"
"but we all know that 12 jellybeans is a week!!"
"No, we all know that 12 jellybeans is explicitly not a week. It just happens that 12 jellybeans has taken 'around a week' in the past. An estimate of 12 jellybeans is still not a promise that it will only take a week."
The right thing to do isn't to estimate the average 1.5 weeks, since the case might not actually be done for another month, and it might end up actually being done by Dev C who just joined the team, for who 1 gold star is currently 3 or 4 weeks.
Engineers are not concerned with time. They are concerned with feature completeness and quality. They do not care how long a task takes, only that it is done Right. When the PM asks how hard it is, that's their only consideration.
The PM only cares about time to make accurate predictions for the rest of the business. A given task takes as long as it takes to be done Right. If the business wants it on a different timeline, the business has to redefine the task, or redefine Right (ie skipping tests or doco).
Research bears it out: human brains suck at estimating time, and are much better as soon as you talk in terms of effort.
BUT if you want to insist on brute forcing through our cognitive biases, I have good news: you can do almost as well by tracking the relationship between "developer time" and "real time". You're still dealing in abstracted units, of course, but you can still pretend you aren't, which is the important thing.
I found your characterization of engineers and PMs not useful and a bit childish. This is not how money is made.
Let’s just agree I don’t understand this. You sound well-informed and I’m just a simpleton.
If the boss wants to know how long things will take, they have to ask the engineering teams. And the most accurate way for an engineering team to answer is with a track record of abstracted unit estimations and how long they took, to predict how long the abstracted unit estimation in question will take.
And when the PHB says "make it two", the engineers can only reply "ok, tell us what features or standards to cut." No different than wiring a house: we can finish faster if you choose some rooms to skip, or some safety standards to skip. But those are the boss's decisions, not the engineers.
BTW I agree with the responder who says quit from those environments ASAP. I don't know how engineers can overcome a dysfunctional business lead like that... and that kind of business lead is often an expert at making sure the repercussions land on someone else. Case in point, by asking you to finish faster he is tacitly asking you to skip capabilities or safety checks... and that way you're responsible for the fallout.
So we just bypassed the charade and starting giving hours/days. The whole team was really just doing hours/days -> points mapping in their heads during estimation anyway.
This is a common story. But because of the calculation step, your PM was still mapping estimated time units to actual time consumed, and using that mapping for predictions. So you were getting some of the benefits of an abstracted unit (accuracy, averaging in unexpected events) without the stress reduction or focus on quality over time.
But if it worked for your team, I would advocate to keep it. Human groups are hard enough to coordinate as it is, there's no need to hew to a dogma.
I wouldn't be at all surprised if the relationship between estimate and actual typically followed a power law relationship, in which case speaking about any averaging over time is meaningless anyway.
Part of the point of estimating difficulty is exactly to reduce the one-sidedness evident in time estimations. Human brains are much more accurate and consistent with effort estimation than time. It's still very likely a beta distribution but for the level of accuracy we're talking about it's just fine.
We should probably clarify the "bar" here: the very best time estimators never beat about +30% delta from actual time. (Note that I'm not using +/- intervals because it's only ever +). So how many points (effort/time ratio values) does it take for convergence to get your effort/time ratio mean within 30% of the peak of the curve?
What's interesting on that specific 50% miss I saw is that the original estimate from the actual teams themselves was argued down as "too big, do it again" by their own management...
A point is a different size for everyone, but a day is a universal (ok technically global) unit.
We deal in time all the time, we know how long an hour is, a day, a week. We can remember "I did something like this before, it took me 2 weeks", not "it took me 13 points".
No manager can predictably translate story points into days, which they need in order to pitch to customers and manage their budgets.
The first thing I ask when I have to estimate in SP is "how many hours is 1SP?" and after a few minutes of the usual back-and-forth, whoever has to actually use these damn estimates always says something like "I treat 1SP as half a day". Bingo, now I can give you a number you can use.
I really don't get this. Can you guys genuinely work 2 weeks on a feature without doing anything else?. No extra meetings, no dependencies, no incidents, no coworkers to help?
Something simple can take me 2 weeks or 2 days - it really depends on all the other stuff going on when I'm working. That's why imo it's so silly to estimate time unless you're tracking every minute you work on that specific thing and manage to seal yourself in a box away from distractions.
The thing I remember when estimating is. Huh, last time I touched this feature it went really smooth. Doesn't look like much work, adding a validator here and I've got an example right there. Factor in some testwork and I guess it's like 3 points?
> No manager can predictably translate story points into days, which they need in order to pitch to customers and manage their budgets.
Either you're a godlike estimator or you disappoint your manager A LOT. Who the hell are these people that can reliably say feature X will take me 8 weeks? Is that also not just an estimate? I'd rather wallow in the vagueness of points than get my ass reprimanded because I said something would take X weeks. If you're looking for absolute predictability you need factory line work.
Basing it on past experience like that automatically takes into account (average amount of) meetings, waiting on others for code reviews, etc.
Sometimes, but you need a progressive team that, e.g. that doesn't do stand ups every day when almost all the work is multi-day efforts.
You also need people who are technically skilled to not bug a developer about something they can find out themselves. More skill = more autonomy = less bother. That holds true for their work too.
A regular occurrence is people around developers leaning on them to figure out technical details because they are not technical enough - that should be their job, so that gap contributes to a higher number of meetings.
>Who the hell are these people that can reliably say feature X will take me 8 weeks?
They're probably overestimating then time the delivery to be around 8 weeks (they are more skilled than their estimate would lead you to believe). A feature like that should get broken up, then you estimate the individual ones. It doesn't matter that the fragments aren't testable/deployable on their own, it helps with the complexity. Decomposing that feature also helps you estimate better.
I agree that if you start giving a single number of days/weeks, that's bad. Then there's a strong incentive for everyone to start padding their single figure estimates to cover their asses, and the managers end up just halving them in their gantt charts and pushing the team to work quicker. That's an adversarial environment, where nobody trusts each other.
> Either you're a godlike estimator or you disappoint your manager A LOT. Who the hell are these people that can reliably say feature X will take me 8 weeks? Is that also not just an estimate? I'd rather wallow in the vagueness of points than get my ass reprimanded because I said something would take X weeks. If you're looking for absolute predictability you need factory line work.
You give estimations in time units because that's the only unit business wants. You can sure say "that will take X points"... nobody will listen to you and they will demand a different answer. It doesn't really matter, actually.
> A point is a different size for everyone, but a day is a universal (ok technically global) unit.
I can definitely see that. Pretty much what you said is what I always hear, but the actual theory behind it (and what I'd guess they're testing in those studies) is that people can agree a given case is the same size ("big" or "small"), but the actual time taken to complete it is going to depend on the individual's experience (both in general and with that specific codebase).
It's just that the time mapping in scrum isn't supposed to happen at the individual level, it's something handled by the scrum master / manager / whoever that interacts with the rest of the business, using an average. This way time estimates when the team has different levels of experience get smoothed out into something hopefully more accurate at the sprint level.
Principles are:
- humans are much better at estimating relative effort than average time. So estimation sessions are only in terms of effort. The answer to "how many hours is one point?" is "as long as it takes." With developers like you this can be a hard line to keep but it is absolutely full stop required.
- consistency in relative effort unit sizes is a requirement for the math to work. Group estimation does this automatically after a few sessions, and can help expose miscommunications and better architectures along the way... but it's not the only way to do it.
- a consistent yardstick for "done" is required for consistent unit sizes. (Logically)
- project managers track average number of points completed per sprint. Even though this average will become extremely consistent, it is an AVERAGE ONLY and can not predict any individual sprint.
- There is no pressure to burn points "faster". Remember, point sizes are arbitrary and consistency is required. When engineers feel pressure to complete more in the same time, consistency is dropped and the math breaks) if you are using sprints this is an easy trap to fall into. More points per sprint != better. Making tasks easier is OK though, ie with automation or technical improvements. Note that this would impact the estimated point size of your tasks, not the number of points put through in a sprint!
That's it. Do it how you want, but have consistent relative point sizes, don't let engineers talk or think about time, don't treat an average like a single sprint prediction, and don't pressure engineers to increase velocity.
A casino can't predict a single hand, but they can predict with great accuracy their profit margin after 100 hands. Using the same math, you can't predict a single sprint, but you can predict with great accuracy 5, 10, or 20 sprints... as long as you help the developers stay unpressured and therefore consistent.
ALL THAT SAID, when you say "I did this before, it took me 2 weeks," that is also a very effective way to estimate. If your tasks are highly consistent, definitely track time for implementation of similar work (because human memory is very fallable) and estimate this way. Just don't let yourself notice that you defined a consistent unit of effort and used past average time to predict future average time, and you can feel like you've found a great life hack.
In principle the “how many hours to one point” question is answered by tracking and analysis of past sprints, where over time you develop an empirical measure of velocity, which gives you an idea both of the average time it takes to do a atory of a particular size, but also the size of the error bars when taking that average as an estimate.
Which can be useful for sanity checking things like sprint sizing, though the existence of the stat risks it becoming an optimization target in some orgs, either because management sees it and wants “line goes up” or for some other reasons (teams can do it to themselves, though management is more likely to.)
I've been through all this with multiple teams/companies and I wasn't always even opposed to the idea! In the end all point-based estimates seem to do is add more ways to be wrong.
Humans are actually pretty good at estimating absolute time if they've had some practice. After the first few times you estimate a day and it takes a week, you realise you're an optimist and you should compensate for that. It seems the wrong conclusion to draw would be "oh well it's impossible, I should use made-up units instead", just get better at it. There was a time I was mostly doing fixed-price work, nobody would have accepted points in a quote and nobody would pay me per point.
Consistency in unit sizes is basically impossible, from what I've seen - unless there's an unspoken but commonly understood translation to units of time.
Project managers who try to size sprints in points tend to eventually give up, after finding that the the actual effort taken varies so wildly between sprints. (I also tend to think sprints suck, and Kanban is the way to go for agile - but that's another topic.)
I've done that too and fixed-price work is like handling dynamite. I've always felt it was good for new grads to spend a couple years in small, eat-what-you-kill, consulting shops to understand the business of software development better. After that, then go to megacorp/faang or whatever but the lessons learned in a small consultancy will help you see the forest through the trees.
The real benefit imo is that it is a good shorthand during estimation that reveals misunderstandings or flaws in a ticket. Everyone pointing a ticket at a 2 is whatever, literally any time spent discussing if it is a 2 vs a 3 drains my vitality, but every time there is a big outlier or bimodal distribution we get a lot of value from talking it out.
I think what points gets you here over hours is that it focuses on the task at hand. I think a time centric approach requires your estimates to be more like 'X hours for John, Y hours for Shirley'.
I'm not touching your points about how there are other contexts where it doesn't make sense to do this, or the absurdity inherent in pretending that they don't still boil down to time in a meaningful sense because I agree with them.
First, I would be interested to see any data you have supporting the idea that “humans are much better at estimating relative effort than average time”. My own experience is humans have biases toward optimism or pessimism and are poor estimators of the magnitude of risk, but I’ll ignore that for the moment and accept the premise as true.
The other underlying premise of your method is that velocity is a random variable about this perfect prediction with a known distribution, such that via the law of large numbers, over a large number of sprints, the average velocity can be known with high accuracy. I see a few issues with this.
One issue is that velocity is not static. Velocity changes with team morale and how interesting or motivating the task is. It changes with how well engineers are suited to a task. It changes with organizational changes, the addition of new team members, the loss of old ones, the work environment, etc. I don’t buy that these are small factors that average out over the long run, either, since my experience leads me to believe they don’t. A poor work environment or coworkers leaving the team or low morale usually are harbingers of bad things to come, and, opposite, a growing team working usually has exciting challenges, morale is usually high, etc. My experience hasn’t shown a mixed bag of equal good and bad, but instead two divergent sets.
Even if velocity were static, the other issue is the distribution is unknown. Sometimes teams have bad weeks, and sometimes they have good ones. Sometimes something catastrophic happens in the organization and a dozen engineers are twiddling their thumbs or are thrown into some task they did not foresee. Sometimes things go unexpectedly well. A priori, there’s no way of knowing what a particular team tends to, or if that trend will continue. One could try to measure it, but my experience has been that something that was supposed to be static will change long before a sufficient amount of data could be collected to be meaningful.
Even if you knew the exact distribution you’d encounter, the final problem that remains is that projects aren’t infinitely long. There aren’t a million sprints to smooth things out. There will be variation in the outcome.
I guess what I’m saying is, estimates are estimates. Why systematize their creation when they will ultimately be wrong, and creating them is so costly? I could understand having a system if it were low-overhead, but my experience with a system similar to what you’re describing was anything but that.
But there is one practical question that has always confused me: how do you "measure velocity" in Scrum, when each sprint's scope, and the story points within, are all preset at the start of the sprint?
Do we assume (realistically) that dev is not completing all stories within the sprint? Or that they are adding more stories in the middle? Or are we assuming that each sprint is finished exactly as planned, on time, and changes in average "velocity" reflect differing levels of confidence or ambition of the dev team as the decide how many stories to include in each sprint?
In a rough KanBan process, I can see measurement of "velocity" as being more straightforward. Am I missing anything?
We get as much done as we get done. We take a guess about how much work that will be at sprint start, based on previous average, and make sure that much work is well understood and prioritized. But tasks turn out to be easier or harder than expected, people get sick, shit happens and you end up accomplishing more or less than what you expected. At sprint end you measure what you actually accomplished, to help predict the long run timeline of the project and maybe to get a better guess at how much work to prep for next time.
There should be no time or "amount done" pressure on the engineers. We measure the actual throughput and use that to inform management how long the work as defined will probably take. If management doesn't like that assessment, they can manage the situation by adding engineers or reducing complexity/scope. If you really like playing with fire you could let them ask to lower quality standards, too.
But how do you accomplish more? Are you in organizations that have enough trust that items can be added to a sprint while it is in process?
Add button in UI = 1sp
i8n text on button = 2sp
New field in database incl backup and migrations = 10sp
And so on.
Once you’ve done this enough times you can correlate it to hours. Never talk about this factor to the ones doing the estimation and always keep the point system fixed to avoid fluctuations in velocity.
That's because they aren't doing it right.
SPs for a team should be measured based on recent past performance. How many hours is 1SP? Let me look at how many hours the team has worked over the last 4 sprints and how many story points they have completed. You only need to ground it when you have a new team.
The problem I see is that people never want to stick to story points and never want to run the calculations. They want SPs to be the same across teams which isn't possible with this method. What project managers should do is look at the feature SP size and the SP per sprint to see how many sprints the works will take, which gives you an equal metric between sprints. You don't say team A is delivering 30 SPs over the next quarter working on a 20 SP feature while team B is delivering 45 SPs over the next quarter while working on a 60 SP feature. You instead look at the features and say that team A is working on a feature that looks like it'll take them 4 sprints to do and team B is working on a feature that will take them 8 sprint to do.
>We can remember "I did something like this before, it took me 2 weeks"
That works if enough of your new work is similar to previous work. I find that is rarely the case.
In "estimating in the small" such as for a development iteration one can often get away without a precise conversion to hours and dollars. But in deciding whether to do ProjectA or ProjectB each being 100+ man-year projects, or whether to reassign the new hire to the team of ProjectA or ProjectB, you still need hours. It's not going to help to say the project is six hundred thousand points, unless you can convert it to the real estimate which is "a 20-40 man team for 4-8 years"
Yes. Did you see the part of my comment where I explained how you do that?
The fact that this is the most accurate estimation method is well established in research. You can get close with some other methods, like tri-partite estimation, but they're more cumbersome.
WHY it helps so much to abstract away from direct time estimation, has a few going theories. One obvious reason is that the abstraction and tracking the relationship to real time brings into play the Law Of Large Numbers, the same mathematical law that allows casinos to profit on games of pure chance. The more sprints you have under your belt, the more your conversion rate averages in unexpected external influences like illness, windows updates, upstream problems, etc, in proportion to their actual occurrence rates.
Another of the most popular theories is that human brains have specialized areas/heuristics for predicting time vs predicting effort, which come with different cognitive biases.
Fundamentally I can't give you a definitive answer to why, only that speaking in terms of difficulty with engineers, and converting to time to talk to business planning, is well demonstrated to greatly improve accuracy in controlled studies. In fact, the best estimators, when predicting time, still predict 130% of the time needed for a complex task. Using abstracted measures it is not unusual to see accuracy within 5% or 10% of actual.
You keep talking about the research, is there any chance you can provide some references? I'd like to review this research as what you say sounds plausible and I'd like to know more.
1. Somebody really needs a time-based estimate for the project, e.g. the customer wants feature X by a certain time and we need to figure out if it's feasible or not in that timeframe, or there's a certain external event (expo, or any other kind of marketing event) with a fixed date and we need to see if we can make it. This is usually rare in traditional product companies.
2. We need to see if the project is worth it, in general. Maybe it's just too much effort to even consider.
3. We need to prioritize a couple of projects/opportunities. ICE/RICE comes into play, and we need to stack rank based on return on investment. Relative effort in meaningless units (tshirt, cups, etc.) make perfect sense here.
Everything else here applies, but don't estimate in time. Estimate in difficulty points, t shirt sizes, cups of coffee, gold stars, anything will do. Then as you track your outcomes (see TFA), measure the average relationship between your units and time. Then use that relationship to project a timeline based on your gold stars or whatever.
Using the law of large numbers to average externalities and developer inaccuracy into your time estimate has been the most accurate method since Kahneman and Tversky's Nobel prize winning studies in the 1970s.