Not surprised this discussion devolved into this bizarro world point so quickly. The developer says “there are too many unknowns to provide any sort of estimate”. The manager retorts “but we need estimates!” As if they expect the developer to say “oh, you need them? You didn’t say that. In that case, the unknowns are now known and it will take four days, three if you remove this inconsequential detail that nobody cares about”.
Unfortunately this bit of illogic is so common that I know it’s a waste of time to argue, so I’ve learned how to figure out what they want the estimates to be, give them that, miss them (just like everybody else does), and then get to work on actually providing the feature.
Seriously though. It's always a time tradeoff. How much time do you want an expensive developer investigating how long this problem will take. Will they learn something they can use during development.
I always follow up my above quote with the unknowns and how hard I think they really are; what are the possible risks; _and_ I include the experience of the developers working on the problem; our ability to bring in other devs who understand this better etc etc. I like to make sure whoever is quoting this problem out understand everything as well as we can in the given amount of time.
If that number is too high, then that’s a sign to management that the work needs to be changed. The only way to determine that is to estimate it then make that call.
And only works if management wants an estimate. More commonly, management wants a lie they can tell the client.
That’s absolutely preposterous. As a senior engineer, I have seen enough that I have at least a high-level idea of what most of the dev tasks and stories I’m estimating are. I don’t always have the specific details that may change the estimate, but I also have a whole team of other people who have seen even more things and can inform that opinion.
The point of refinement sessions is to share this information with everyone else so an estimate can be created and if needed, tasks can be better defined, split up or merged.
> And only works if management wants an estimate. More commonly, management wants a lie they can tell the client.
Wat? Estimates is used for plenty of reasons that have nothing to do with external clients.
How do you think your manager, their manager, etc knows what is going on and approximately how long it’s going to take? They’re going to be far more disconnected from the work than the people doing it day to day, so they would generally have no clue how long something is estimated to take without someone telling them.
You are misunderstanding your job. You are a subject expert and paid accordingly. Answer with what the unknowns are, an optimistic and pessimistic take on them, what you could do and how long it would take to lift them.
If you don’t, your boss is going to ask you for exactly that in a roundabout way and micromanage you because you are forcing them to do your job.
It still baffle me than approximately half of developers don’t understand things which are obvious to any seasoned blue collar worker.
In software, on the other hand, we often have most training being taught by professional academics who do not understand and are not qualified to teach the parts of the job that don't have a straightforward academic analogue. Combine that with a business environment where things like budgets and cost control are often only theoretical, and all but the most gross liability for defects due to poor workmanship has been waved away by EULAs, let it marinate for several decades, and... yeah. Not only do people not know these things, but I'm not entirely clear on how or why they should be expected to learn them, either.
If he really needs an estimate he'll tell me to start digging into it right away, and I'll let him know after a bit of time how it's going. If there are still large uncertainties and my boss still wants this completed by some specific date, we might try to explore alternative solutions.
In the above case, I didn't get that time I would need, they wanted the estimate to include that time I would need to dig into it. But that's above my estimation powers. I didn't even know if what they asked was possible at all.
They went and asked someone else who did guess some number of weeks, but the work never got priority so we still don't know.
Yeah that ain't gonna fly with me.
Did you say that
1) Without knowing further best case is it takes N time, worst case M time, and if there are certain obstacles it could be completely unfeasible. 2) To get more accurate time and to understand whether it's feasible I would need O time to figure that out first, and I can give you the more accurate estimate then.
The problem is when we assume everything is a known. I try not to do that and I try to set that expectation with my manager. If everything had an easy answer we wouldn't need highly paid problem solvers.
Well, that's the problem right there. An estimate is not a promise. It should not be treated as such.
It's not an apples to oranges comparison. You give me an architect's blueprint and guys who build to the legal building code, and I could probably get a house built. Expensive and painfully, but it can be done.
I don't even know what the coding analog to an architect's blueprint is. They are fundamentally different jobs.
Yes, I work as a delivery lead nowadays. Producing and respecting estimates for software projects is literally my job.
I routinely fire developers who pretend they can’t do estimate and strangely the ones I keep are actually not only able to do it but good at it.
I've met your sort before. You're a net waste of resources in an org: instead of developers getting things done, they're dealing with you asking them to do your job for you, i.e. produce estimates for the things they'll never actually get around to doing (because of producing estimates).
Your username would be accurate if s/My/Everyone's/
If it helps you sleep at night, sure, assume I’m a net waste of ressource. Good luck fighting with the other seven teams for some time with the shared senior UX expert.
If you rarely eat your time buffers, great. If you do that often, maybe you just coinflip-fix deadlines this way and maneuver between them under the guise of successful estimation.
You always get what you ask for. Like sure I can estimate. If you ask for a web server you'll just have that. It won't be secure. It won't be fast. It may work. It will certainly pass the tests you think of.
Is that actually good? No. The real estimates get rejected. Everyone cuts corners. When something does happen e.g. a hack occurs the engineers not these managers get fired :(
My optimistic/pessimistic takes at my last job would look something like "roughly a week to roughly 2 years"
Depends on how deep the yak shaving goes, and you wouldn't know until you got halfway through fixing the problem.
Often I could take an educated guess as to which side of the scale it came down on, but I've definitely started pulling on a string inside of the codebase and it just kept going and going and going. All of those baked-in assumptions that other developers unconsciously made over the course of a decade can pile up into a mess that you just can't implement the feature/bugfix without doing a pile of potentially breaking changes that need to be shipped in major versions.
Generally if I worked the problem for a week then I could give you a much better estimate of when it would be done, and often that estimate was "it's done now" after a week. But if I hadn't looked at that particular subsystem for a few years, I couldn't trust my memory of it up front and had to do exploratory surgery first.
(Really exploratory surgery is a much better metaphor for what you're going to be doing, and surgeons can routinely find surprises -- "oh look that barely perceptible shadow on the x-ray turns out to be a stage 4 tumor...")
Yes, the codebase was horrible, yes it needed to be cleaned up, no we didn't have the manpower to do that and it would have taken me 10 years to rewrite it all.
Quitting of course was an excellent solution.
You're lucky if your management chain allows you to work on problems for as long as a week without giving them an estimate first.
Add a small feature to an existing system. This estimate can be anywhere from several months to a day. Most of the error bars are org and system specific rather than anything to do with the feature. The exact same feature built by the same person in different orgs and different systems can be opposite ends of the range.
Seldom do people have the contextual understanding to provide an accurate narrow estimate, particularly since both org and system are in flux. They change constantly. Unseen landmines potentially await in both.
> If you don’t, your boss is going to ask you for exactly that in a roundabout way and micromanage you because you are forcing them to do your job.
And such orgs take forever to get anything done.
> It still baffle me than approximately half of developers don’t understand things which are obvious to any seasoned blue collar worker.
Software is probably the hardest kind of engineering you can do. All the system constraints are virtual. Just consider comparing a dirt ground for consistency and behavior vs libc... 304 stainless vs react.
You can go deep in both, but a shallow understanding of one takes you much much further than a shallow understanding of the other.
That’s extremely untrue in my experience. Sure, hard to interface with systems are a thing and organisational roadblocks happen but an onboarded senior team member can tell you what’s going to take days and what’s going to take months or they have no business being senior in the first place.
I feel like my answers in this discussion have been particularly harsh but I would really like people calling themselves software engineers to start acting like some. I think my view on this subject has been strongly shaped by spending most of my career doing software development for industrial companies: people who claim software is particularly complicated just don’t understand and respect how complex the rest of an industrial project is.
You're making an unwarranted assumption here. There's many pieces of software that have no development team at all. There's no one to ask; the last person to touch this software left the company three years ago. All they can tell you about it is how they use it now.
It's likely there's at least someone that can explain their business process to you, but identifying them and receiving enough of their time to piece the picture together is another story. And maybe, if you're lucky, that person can tell you what the business process was at the time the software was written, and what has changed since then.
But this process can take weeks, and most of the time you're expected to provide an estimate before even talking to anyone from operations.
Sure, for internal software with an active maintenance/development team, informed estimates can be given with a relatively high level of confidence. But as an outside consultant working with non-software companies, all estimates are completely made-up based on your reading of the buyer's budget: there are no other data points to base your estimate on.
Development by outside ressource happening on custom pieces of software from non-software related companies is such a niche it might as well not exist. I don’t think it’s a very useful datapoint in a discussion about estimation in software development. Plus, as someone contracted, I’m fairly sure you actually have a rough idea of the effort involved or you would have become insolvent already.
And yes they are still solvent.
As others mention there are lots of unknowns in an organization.
I've been in places where the other team involved is across the globe in a different timezone and isn't even actively willing to respond and yet we continue...
There are also cases where lots of parts are outsourced and you're lucky to get a reasonable response i.e. changing part x requires vendor a, b, c & d, which have an SLA 1-8 weeks and so you're already starting 4-32 weeks without even working anything else out. If you're really lucky it could still all be done in 1 day.
Not sure if you're harsh as you say or just lucky. Not everyone and every team is well organized.
Yeah, most of the time my boss just needs to know if this job is a few hours, a few days or a few weeks. I never answer with exact numbers, but a range. If there are some major unknowns I let him know. If I made some important assumptions then I let him know.
Sometimes I'm wrong. Most of the time my estimates are sufficiently accurate.
Sometimes the customer really needs this done before some deadline. Then it often turns into "how can we make this happen before then". After a bit of back and forth with some added assumptions and cutting functionality, we usually end up with a solution that can be realized in time.
Assumptions can range from “if this API can be consumed in a usual manner and is well documented” to “if these clarifications are made in time” to “if the domain experts test the feature on our dev environment and give timely feedback”.
I try to communicate clearly that it’s almost always a two way street.
Yes I’m the technical expert, but they are the domain expert. I need their understanding and feedback to make something that’s useful.
The best clients understand this very well. Those also get priority and sharper estimates.
As a manager now, I get scoffed at if I even hint at a deadline. Excuse me what? We have a limited budget, a limited runway, and we need to prioritize which product bets we are going to make. This seems entirely logical, but a lot of devs I work with now find it impossible. They just enjoy tinkering away with whatever thing interests them, and delivering value only means what they personally find valuable.
Personally, I find it a bit pathetic and childish. I say this as a long time software engineer who knows perfectly well how this business works.
That said, however, The problem I run into during practice is dependencies— people want estimates to integrate into systems you have never heard of or used before. Not only that, there is no in place architecture or design before estimation. Imagine a construction company estimating a house without knowing how big it is or without having blueprints.
Often times software development is designing and architecting the system — not just implementing it. And that is the expensive part.
Say that the estimation can depend on the complexity of the system. I would think worst case is when the system has these A, B problems, then it would take N time. Otherwise best case it could take M time. If we want more accurate idea, we can spend O time on it to figure out what it could be.
How does this business work?
Why is software special? This happens the same in construction and other industries. The estimates usually make 0 sense and there are often huge delays. Same as store openings, film production, etc.
There's a problem with tinkering and devs going off in the wrong direction but on the flip side when people say that something is done in an hour or a day often never happens. Managers and agile practices that want tasks broken done to the smallest atom just messes things up. To make things work product often never has accurate requirements to that detail either so that complicates things even more. The requirements change on the fly.
In a healthy culture it should be possible to have a back and forth about how accurate an estimate is likely to be, but when engineers insist that all estimation is useless you can imagine why other stakeholders may lost trust.
As a developer I never get “that’s a useless estimate we can’t prioritize based on that/tell the customer that”. It’s a perfectly good estimate. It tells everyone there is a lot of uncertainty. The low lower bound indicates there is a chance it might turn out to be cheap (so maybe worth taking a gamble and trying it and giving up). The big variance indicates it could/should be given some time to prep/investigate in order to narrow the estimate.
When there are unknowns, make a wide estimate. Convey the uncertainty. If the unknowns are ridiculous, make a ridiculous estimate. Even “5-200 days” is useful because something else with less uncertainty can be chosen instead.
For context, my team is in exactly this situation with several silicon bugs we found last month.
I'm running through a similar situation now, where I'm somewhat adjacent to a client and a dev team. The client has a list of tasks he wants done, and he'd like banded estimates for the effort the dev team thinks it will take to complete each.
The developers are obliging, but one of them complained to me privately "why doesn't he just give us 2 months of budget and we'll get through as much as we can?" - which is an interesting statement. From the developer's perspective, this is ideal. They are very good and work hard and have integrity, so I'm sure something like that would work, from their perspective. They can just 'go' and get stuff done.
From the client's perspective though, it just raises a bunch of uncertainty. He wants to know what he could get out of his 2 months of development budget. If one task took all 2 months, and the others (15 in this case) didn't get done, is that OK? What if some important task wasn't that much effort? You'd potentially want to priortise that over a bigger item.
Just saying 'give us two months of budget' doesn't take the other party into account, and what they're trying to balance or plan for. I've sat on both sides of the table, and honestly just asking for a budget with no actual commitment to completion feels selfish in my opinion. Discounting their opinion because they're 'business' people and not developers, or treating them like an enemy to fight against won't win you any friends or long term relationships.
The irony is that at least some of the time, a "safe" estimate will actually make the project take longer. Work naturally expands to fill a deadline, and not having a deadline but instead frequent progress checkins with an expectation of foward progress can result in more getting done, because it changes the conversation from "how at risk is the deadline" to "how much happened in the last week and what are the next steps", forcing progress to happen continuously rather than one big bang at the end.
I also often hesitate to give estimates / establish deadlines because it tends to create an unhealthy confrontational atmoshepere, where developers are incentivized to not incorporate feedback or make any changes even if they are the right thing to do, because it would put the deadline at risk (but if they incorprate allowances for these things during estimation it's considered sandbagging).
That’s probably the most important thing I’ve taken away from software project management techniques - everything they do is hurting them, not me. If somebody insists on shooting themselves in the foot, and they’ll shoot you if you try to stop them, you just sit back and let them shoot themselves in the foot and shrug your shoulders when it happens.
At the end of the day, these estimates are helpful but maybe not in ways that are intuitive from a developer’s perspective. They’re useful in weighing priorities, judging the uncertainty, and identifying where to fill the gaps in understanding, time-boxing research and scope (e.g. do we really need to spend 6 weeks reading release notes?), and help communicate to outside stakeholders. We see the successes of this all the time with big game releases, product launches, movie releases, etc. It’s a high level measurement of progression and devs need to be a part of that planning process. It gets easier for senior engineers to do this efficiently with experience. It’s not necessarily an exercise that a more junior engineer will a) get right consistently or b) think is valuable at all.
1. Coordination of external dependencies. Stuff like product marketing, sales training, documentation, etc. as well as sometimes things external to the company like customer commitments or sales commitments. These are for the most part legitimate, although in many cases the deadline doesn't need to be known at the outset of development, and too many of these is a very unhealthy sign.
2. Accountability / improved productivity - This usually manifests as an exec feeling that engineering is running too slowly / inefficiently, and by asking for deadlines (and often compressing them to "push" the team) they can improve productivity. These are almost always in my mind completely counterproductive, and there's much better ways to accomplish this.
It may feel good to have some of the times quantified. But... If your estimates are literally fiction. It won't stop being fiction in front of the client. You should really spend that energy that went into the estimate for something useful instead.
Not to say you can't derive some useful information from a short planning session. Just don't expect anything other than thumb to the wind guesses.
Maybe the solution here is "stick to your guns", but that can have negative career implications.
You can’t pull a different estimate out of thin air. And zero people will expect it.
The developer needs to say “I can say 10-100 days right now in this meeting, but given just 1 day of investigating, I’ll narrow the estimate range by an order of magnitude”.
As a PM, I couldn't possibly agree more with this. When you give me 5-200 days instead of "we can't estimate this", you've started the beginning of a productive conversation. I can ask you questions about why the range is so big, and you can explain to me what steps we can take (research, narrowing scope, etc.), to reduce the range.
Ordinary LoB staff apparently are superstars that have their schedules locked in months into the future, but consultant SREs costing thousands a day can have their time wasted on a daily basis trying to determine how long it will take to complete a task that hasn’t even started yet.
This has literally been my last two months. The dev team lead is asking me — an outsider — how long it’ll take his team to finish something. Not because I know better, but because he doesn’t know either and prefers to make it my fault because I’m expendable.
1. Talk to the person who'll get angry at you when the UAT testers are brought in at the wrong time.
2. Make it clear to them, with a paper trail, that this is an impossible task and you can either book in a best-estimate with the likely outcome that the UAT testers will arrive and there will be nothing to test OR that you can book conservatively and there is a risk the project will be delayed waiting for UAT. They can pick which risk they want to bear.
3. Put in formal feedback, as loudly as possible, that booking in UAT for a specific time introduces obvious risks and someone needs to optimise the process to be more flexible otherwise it will delay projects & waste the businesses' money.
If you're the user or representing the user, shouldn't you be doing the UAT?
You can pull a different estimate of 20-40 out of your behind when pressured to narrow it, and make the deadline 40 days. That might be a 50/50 hit/miss the deadline, with a worse case of running over to the full 200 day worse case (or even worse - even the 200 was a guesstimate). In some cases that's fine too. The decisionmaker might just want a date with 50% chance of hitting because it's better to land a sale and apologize for the delay, than to not land the sale and go bankrupt.
The correct course of action though is to not set the deadline in that meeting. The 5-200 day estimate is a way of explaining "Either don't take this on at all, or at least don't try to schedule anything urgent based on it".
In that case as a developer I'd suggest "Let me look at this for one afternoon, and I can return with an estimate that's 1/10th as large. Then let's meet again ad prioritize some work, schedule UAT and so on". PM's often look really surprised when they realize that all it takes might be to check if there is a functionality available already or if it needs to be implemented - which represented the difference between 5days and 200. If I go off and find it's already there, the estimate is 5-10, if I go off and find there is nothing and it has to be built from scratch, it's 150-200. But without that knowledge: 5-200.
Or, if a 20-200 day estimate is actually due to real implementation uncertainty, I'd suggest "This is potentially large and uncertain. Let's try to break this apart if necessary into something that has a 10-20 day estimate and some parts that have a longer horizon. Schedule the delivery/UAT of the first functionality in 20 days, and we'll by then have a good estimate of when the remaining work will be done".
If someone in my organization would try to convince me to pull a stupid estimate out of thin air, then pressure me to haircut that estimate, then schedule something based on it (such as a delivery), then pressure me to work overtime to meet it, then one of us would be leaving the org very soon.
I quite literally quit my last job in February because of this shit[0]. Stop. No. These are not useful things for developers, much less senior developers, to spend their time on. 95% of my job ended up being coming up with estimates. My top deliverable to my management chain was estimates. They would only care about the actual development work insofar as it validated or invalidated the estimates deliverables.
Hard facts: If time is such an unimportant resource that you're willing to waste most of it doing "research" and worrying about scope, then you don't actually need an estimate. For reporting purposes, just BS one yourself, that's literally your job as a PM.
[0] It was well-paying, I was basically fucking around every week doing whatever I wanted, yet this issue alone outweighed all of that and was enough to make me quit.
I do that and I've become exceptionally good at blaming the business persons for the outcome, making them admit their guilt in public and all that in what seems a friendly, positive and constructive manner and using business motivation and explanation. They never understood what hit them. Every programmer can do that if he or she learns a bit of psychology.
If some manager made a promise to a client out of his/her ass without asking us, and if the PO pressures us towards meeting an unrealistic time-frame, when that explodes I just explain in a public meeting where upper management and all stake holders take part why it was wrong to promise that, using arguments and business considerents because bussineses people don't care about tech considerents. It usually results in that person admitting in public their mistake and even saying they are sorry.
I don't do that to humiliate people but to teach them to ask the team implementing the feature to estimate or to say if that feature is even reasonable or possible given current state and resources.
The estimates aggregate to a number more accurate than each individual. The manager cares more about the sum total than each individual one. Many can even slip without affecting the end date.
This isn't so hard, if management and developers didn't treat each other as enemies. Each can get wrapped around the axle concerning each deadline, rather than the real goal of allocating resources across the entire scope.
The difference is, the law says if a contractor signs a contract to do something at x price, they HAVE to do so or suffer consequences. So contractors go broke/bankrupt, lose their licenses, or figure out how to bid correctly enough they can either weasel out of it if it goes sideways or they have enough wiggle room to deal with the inevitable problems.
Software Engineering Consultants have the same situation. But directly employed software engineers are in a different position of course.
One thing that typically makes this worse, is the implicit assumption that planning/quoting/estimating is ‘free’ work, and since it CAN make things easier to deliver, therefore we should do more and more of it as, since it’s free, it will solve the problem no?
And their disciplines involve massive capital outlays which can’t be refactored in a cost effective way, or at all really.
Not defending the ‘no estimate ever’ approach, it doesn’t work. They are different though.
Tell me, how accurate were the estimates for the launch date of JWST?
I’ll wait here while you look up how many times and how many years that “slipped”.
I bet some dumbass NASA manager asked the team to schedule the launch date back in the 1990s.
It gets even worse when you are in a line of work where the estimates are part of some kind of sales pipeline. Then you aren't just estimating based on effort required, but also against a likely number that the customer will ever agree to.
However, I don't think this idea would be very popular because it removes the manager's ability to control the presentation and play politics.
Except instead of anonymity, the devs have to actually discuss until they reach consensus.
The correct solution here isn't just to insist estimation is impossible and be offended that someone would ask for it - that just misunderstands how businesses work.
The correct thing is for either of those parties to say, "Okay, I understand it's not estimable at the moment. What are the steps that we need to take in order to get a reasonable estimate?"
A good PM will do this, but where there is a bad PM, a developer can step in and say the same thing. In your scenario, "The developer says “there are too many unknowns to provide any sort of estimate”." What the developer could say is, "There are too many unknowns to provide any sort of estimate - let me give you a few examples. <examples>. What I can do is come back to you after <amount of time> with a list of the unknowns and an explanation of the work required to evaluate them, so that we have enough information to estimate. Once you have that list, then you can decide if we want to proceed with the work to understand those unknowns."
I totally understand why developers get frustrated when they're asked to give estimates for things that can't reasonably be estimated with the current information available. But the flip side is it's also very reasonable for PMs to get frustrated and ignore developers who insist that estimation is an impossibility, because that's virtually never true - maybe it's impossible to do now, but there is a path to make it possible. If you work proactively to expose that path to the PM, it'll generally result in everyone being happier.
Sometimes we don't know what we don't know. If we don't know what we don't know now and if we don't know what we'll find we don't know later, then we don't know how long we will need until we'll know.
Uff. Sorry. If I had more time, I would have written a better comment.
If it's the latter case, then a good developer can describe the immediate steps they'll take and what questions they expect to answer. Whatever the case, there is always some concrete next step you can take to which you can assign a reasonable estimate of time. That might just be that you'll spend one day perusing some documentation, and the only outcome you expect from that is that you'll develop a reasonable enough understanding of how large/complex the documentation is, such that you can come back with an estimate of how long it will take to find and carefully read all the relevant sections of the documentation.
That's totally fine - if it's the sort of wildly unknown problem that you're describing, then I recognize we'll have to take steps to understand the problem. There's still never a case where it's impossible to create an immediate plan of action and understand roughly how long that will take (unless you are simply unqualified to take on the problem).
I had a boss loved the phrase, "a date for a date." He just meant that if you can't give me a date for when the thing will be done, then you need to give me a date for when you can give me a date for when the thing will be done. Can't do that? Date for a date for a date. However many levels we have to go, you're not getting out of whatever meeting we're in until you can give me a date by which you'll be able to come back and give me another date.
Right now I’m replatforming a legacy app. It’s undocumented, developed by dozens of staff — all of whom are gone, and full of crazy bugs and gotchas.
Out of the box it wouldn’t compile, the source code was missing files, etc…
The task is to find every such hidden issue, fix it, which then allows us to find the next hidden issue, and so on.
When we’re done finding issues, we’re basically done.
Can you tell me how many issues we’ll encounter? I can’t. I’m being perpetually surprised by the code base, finding random binary blobs that actually are from some other project we didn’t even know exists until that moment.
Fundamentally, it’ll take as long as it takes. The application is mandated by law, and it must be made to work. It’ll cost what it costs, and it’ll be finished when it’s done.
That's exactly it. Scoping out the solution is 90% as hard as just doing it. By the time you've figured out enough of the details that you're sure there are no more surprises waiting... you're just about done.
I suspect that a lot of counter-examples people have witnessed are examples of poorly factored codebases. I've seen teams/projects where they estimated projects accurately because the "new" thing being created was a slight variation on a kind of thing they'd created many times.
For example, I once worked for a company that generated e-commerce sites for clients. They'd make a new one basically by copy/pasting an existing site from another client and changing some text/images here and there. Super predictable after doing this a bunch of times... but also a maintenance nightmare having dozens of slightly different codebases. And how do you iterate? They couldn't evolve.
Shopify ate their lunch. I bet Shopfiy has a really hard time estimating the time to ship a new feature. But they add it once for everybody.
I love deadlines. I love the whooshing noise they make as they go by.
- Douglas Adams
A knowledgeable developer can predict how this cycle goes and explain that an estimate of the whole task is impossible, and they will be quite literally and completely correct in that case.
Ah, that would be nice. In my experience it's more like, "Okay, I understand it's not estimable at the moment. Just make your best guess and we can revise it later. (Psych! I will hold you to this estimate like you were under oath in court!)"
This leads to things like the Scotty rule. Since management will always misuse, misunderstand, or misrepresent your estimate anyway, might as well look out for number one and inflate the estimate, just in case.
The only time I've really had management "misuse, misunderstand, or misrepresent your estimate" is once they got past the point of being several hundred people with the bureaucracy that entails. Then I leave and go to a startup that's <50 people.
In my experience good communication solves this every time.
MGMT: We need an estimate ENG: I have several unknowns MGMT: We really need an estimate ENG: This is my rough estimate, but be aware that the unknowns could change this drastically MGMT: Ok, what do you need to get a firmer estimate.
And so on.
As a project progresses, the communication between ENG and MGMT is crucial to ensure that MGMT are aware how those unknowns are affecting the project, MGMT will decide at every juncture if the project is worthwhile to continue to invest resources into it (assuming they avoid the sunk cost fallacy)
As an industry we just seem incapable of coming up with any other way of negotiating and running projects.
This would allow for planning, prioritising and is the most accurate communication we can achieve.
Because that’s what we do either way. Round numbers from ass make no science to plan and prioritize upon.
1. Assumptions (including soft such as who will actually do it, resources are not equally relevant or productive) 2. Constraints (as agreed by stakeholders) 3. Risks 4. Clear scope 3. Alternatives and why they were not preferred
Business should be aware and own the uncertainties, or else the system gets biased towards sales people offering vaporware.
HOW developers should make their estimation is a totally different question, but it is not some arcane knowledge, and it should be an everyday craft for every mid-level engineer. First of all, we are doing those tasks by hundreds through our careers, so an easy way is to remember how long similar tasks took us last 10 times an produce a bracket, like in "from 5 to 8 working days, let's say 10 to be sure". Every business person will be happy with that answer.
OP says that every engineering task is unique as a piece of art. This is BS, IMO, except for the very junior engineer. Most of the time business wants some well-known and hundred times done things they see their competitors do. All the uniqness stems from particular codebase. If your codebase is a mess - you are in trouble, but it is you problem, not the general problem of software engineering.
In case I have some really unique task, what I should do is to say: "I cannot estimate this right now. Give me two days and I will get you an estimation". In those days I decompose the task into smaller, and divide those smaller tasks into "well known" and "unknown". Then I imagine (or even draw) a trivial Gantt chart and will have my estimation.
It is also totally normal to change your estimation during the work. You just should do that the moment you see a problem, not at the day of the deadline.
Op is not saying that having estimates is useless, but rather it should be a management task rather than a developers burden.
To me, that sounds even worse, but I do think devs are expected to do wider and wider tasks across the business. Their post reeks of resentment against management and PMs and I don't blame them. Especially in my experience, I have worked with PMs that I have no idea what they do as devs need to step in to do all project management related tasks themselves
Asking the builders to estimate the time they need is a good sign of high trust in that team. It’s healthy. We should want more of it. The business’ side of the commitment is to understand that it is an estimate and should be treated accordingly.
I was tasked with creating an entire email campaign system (like mailchimp). But I had only 3 weeks to do it. I made the deadline no problem, but the system had numerous obvious issues.
My boss didn't care, because the task was not about delivering a perfect email system. The task was getting something useful in the hands of customers rapidly.
If a feature starts getting heavy use, then more investment is justified.
That's where it all breaks down. What does "treated accordingly" mean? In reality it means the estimates can't be used for anything because they are always unreliable. There is no such thing as a reliable software estimate, ever, unless what you're doing is so repetitive that you're about to be replaced by AI.
But that's not why people ask. Instead, half the time you give an estimate it gets immediately turned into a deadline, even if you were promised it wouldn't.
The top comment on this thread is naive. The reason devs hate giving estimates or point blank refuse is because it is meaningless, that's not how the software business works. Analogies to plumbers and builders just reinforce the naivety. Guess what, blue collar work is only predictable for as long as it's highly repetitive which being physical in nature, it often is. The moment what you're asking for is "build me an underground railway using the latest technology" it turns out construction estimates are worthless too (see: Crossrail).
One reason tech firms crush their competition so reliably is they don't have an estimates-deadline culture, because they're run from the top by programmers who understand intuitively why they're pointless. Instead developers are incentivized by equity, bonuses etc to do the job as fast as possible.
It’s less that I now have a specific endpoint, but rather I have a target (or multiple targets) at which I can decide if we’re on track and, if not, how I may want to change previously made prioritization choices.
If you tell me a day, then 5 days later if it’s still not done I may not rip you for being wrong but I may decide it’s time to pull the plug on that initiative since it no longer meets the criteria under which we agreed to do it.
I’m all ears for another proposal, but “just keep building stuff with no regard for how long any of it takes” is not much of a strategy and doesn’t reflect the way that decisions have to get made in the real world with limited resources.
Also and I cannot stress this enough, if you can make any estimates at all you're playing on easy mode. If the stuff you're programming isn't entirely rote boilerplate then you cannot estimate anything, ever, because your error rate will be measured in orders of magnitude not multiples.
Classic scenario: "how long will task X take?" ... "hmm well there's an open source library that claims to do X here, looks pretty easy to use, so maybe two days?". Some time later: "why is X taking 2 months and not 2 days?" ... "it turned out the library has some non obvious architectural limits we can't fix and which kill performance for our use case, so we have to write our own from scratch".
That sort of thing - you thought you could rely on another component and then you can't - can happen at any time, to anyone, and renders all estimates based on the assumed viability of that component meaningless. But you're always building on third party work. If you have absolute confidence in all your dependencies it, again, means you are doing repetitive work that isn't much different from your last task. Such works exists, I guess a lot of frontend dev falls into that category, but it's also highly susceptible to being automated away by better frameworks or AI.
>> it doesn’t reflect the way that decisions have to get made in the real world with limited resources.
My experience was the opposite actually, the big successful companies do exactly that. They may cancel projects if they seem to be no hopers, but they don't try and estimate up front how much a project will take because they don't think in terms of projects at all. If something is important, it gets staffed until it either stops being important or the team is beaten, at which point it gets unstaffed. If progress isn't fast enough they add more headcount. Everything is thought of in terms of relative resource flows, not timed Gantt charts.
This seems like throwing out the baby with the bathwater. An unknown thing can still have bounds on it: I don't know exactly how far it is to the nearest supermarket, but it's further than 50m away and closer than 1km away. That's useful information: I know I'll be able to walk that distance, I know I can probably carry a reasonably heavy load back, and I can roughly estimate how long I'll be gone.
Fwiw, I agree with you that estimates are too often used badly by management to set up deadlines rather than genuinely make planning decisions, but I also think developers often fail to communicate their confidence levels when giving estimates. There is a material difference between "I know exactly what the problem is and what needs to be done, this will take two days" and "this looks like a pretty minor issue that will probably take two days, but it's touching a system I'm less familiar with, so it could end up going on a lot longer if strange things are happening there".
The best combo project/product manager I had didn’t have either official title, but we were on an internal team and somehow she always managed to make sure people around the company knew what we were doing and delivering, pester people so that no tasks were dropped, etc.
Interestingly, she did most of this by putting tickets for projects into Google docs, and otherwise didn’t buy into any of the standard project management dogma. She had an ear/nose for who might want to use our stuff and was relentless at getting us on internal presentations of any teams/groups that would be interested.
Count how many people are also complaining about their managers giving timelines to clients. Damned if you do, damned if you don't.
Also, a job on his home is exactly what it should not be compared with, since that's the sort of repeatable job that can be estimated.
To some degree and depends on the job. Start opening up walls in an old house and tell me how cookie cutter it is. When I used to run shipyard jobs on offshore drilling rigs, I can tell you that every single job ran into all sorts of problems that were individually not predictable.
It's also just nonsense to think that most software is this uniquely open-ended snowflake. Most engineering jobs require problem-solving that can be affected by all sorts of things in the physical world--plus you sometimes just have to build and try out designs and maybe they work, maybe they need to be refined a bit, and sometimes you have to toss them out and start again from scratch.
I see people on here post construction projects as the poster child for things that can be estimated with traditional project management techniques. This is wrong because most construction projects come in over cost and over schedule.
This can be because materials costs change, labor costs change, the soil at your site may not be what you expected, weather impacts, and stakeholders may demand something different after the project is estimated.
Although these are unknown quantities, these factors existence are known. Whereas in software, you’re more likely to encounter unknown unknowns. These are factors that nobody thinks are even possibilities until you run into them because nobody has done anything similar before. An example from my experience is when I was working on implementing a marching cubes algorithm on a GPU, we did research to make sure that you could do it. But when it came time to implement, we found that the cube data was expressed as a set of faces, and not a set of vertexes and no efficient algorithm existed for that expression of the data that could be mapped to a single instruction multiple data paradigm. We could not preprocess the data to convert it to vertexes because it would not offer any speed up if we did. To implement the acceleration we wanted, you’d have to change how the data was stored in the application. Which is a very different project with very different timescales. We ended up abandoning the effort and moved to other work. This happened because we made a discovery that no one could have predicted, in the implementation, that had an unknown impact.
In physical projects, you might find there are a lot of unknown things that can happen, that you may not predict. You can certainly reduce this by bringing in experts to a project, people who have seen much more than you. Certainly there is still potential for black swan events. But these are rare. Where as in software, the experts you may bring on may not necessarily know more than you, you are frequently your own expert. Black swan events in software projects are common.
But they were surely predictable in aggregate, no? There's only so much volume in the house and a volume will take a bounded amount of time to deal with, even if the bound is very large. An oceanic vessel has a maximum amount of mass that can be added or removed, and every unit mass will take a bounded amount of time.
This isn't the case for development. There is a bound to how much code can possibly go in a project, but there is no bound for how much development work goes into any unit of code.
Only in the sense of things always take at least twice as long as they "ought" to. The volume of the vessel has very little to do with it. In a refit, you know some things going in--which may indeed be relatively straightforward (so long as the right parts arrive on time)--but for bigger jobs lots of things crop up including resource conflicts and parts that need to be ordered.
If software projects added more slack, they'd be more likely to be completed on time too. There's nothing magical about software. And, in fact, software doesn't need to deal with a lot of the quirks of physical systems.
Vendors and even open-source libraries are often liars or just missing what seems like it ought to be obvious functionality. Sometimes tools are just broken. I once was assigned a Java AWS Lambda project, which, great, supported, right, says so right on the website? Except half the CLI tool functionality just didn't work, at the time, for Java projects, and I mean the official tools. It was alpha-tier supported at best. Usable, sorta, kinda, but you're gonna lose some time un-fucking the situation, and the whole thing would be slower to develop than it needed to be. You'd not know that unless you went and did a focused search on their issue tracker(s), which isn't really reasonable to do for all the stuff you might use in a typical project (some of which you might not even know you need until you've started work). "Great, so now you know that for estimating next time, if you get assigned a task exactly like that one" well, no, not really, because that was like four years ago and the situation's probably totally different now (maybe better, maybe worse, who knows?)
That's why I think giving 4-8 weeks of development time with some basic t-shirt sizing or pointing for tasks and just seeing how it's going is better, before estimating. I think if you can't afford that, before getting the estimate, you can't afford a half-decent estimate at all and just need to deal with the fact you're going to get a terrible estimate with an absolutely huge range ("six to eighteen months"). Which, fine to ask for that up front then refine it after some dev time! It just needs to be clear and expected that those early estimates are going to have some wild variance—but they can definitely tell you "is this more a 1-year project, or more a 5-year project?" which does have some real value.
I know many developers enjoy the art of software development and want to continually improve it, but that isn’t always the task you are being paid to do.
There are many ways to remodel a bathroom. Some more optimal than others.
Now imagine your plumber tells you he discovered a better layout for the plumbing just after finishing his work, then he enthusiastically asks if he could tear everything out and start over (and pay him for it).
No, just all software that provides any sort of tangible benefit to anybody. I’m sure you could estimate fairly accurately how long it would take to implement quicksort - but nobody’s going to ask you that.
It's not strictly correct that most software is open-ended research. Most economically valuable software is open-ended research, because otherwise a solution already exists and you don't need to pay a software developer to write it.
Software is a somewhat unique industry in having zero cost of replication. The only other industry like this is creative & artistic pursuits (writing, artwork, music, film, etc.). That industry is also notoriously difficult to estimate and tends to be an area where the the greatest works of art are done when they're done. (How long have people been waiting for George R. R. Martin to finish A Song of Ice and Fire?)
> https://www.computer.org/csdl/magazine/so/2011/06/mso2011060...
In some cases I’ve got agreements up front that if the dependencies aren’t done or being worked on by particular dates my team gets access to the codebase or systems and can take over. That type of consequence/potential embarrassment can set up potential adversarial positions but it has helped a few times.
They key is negotiations, and both parties being able to balance a moving target of requirements and scope. Large software development is not a determinable thing as much as we would like. Perhaps ChatGPT will solve this?
>> I wonder if the person asking the question would be happy to let someone do a job in his home with an uncapped budget and timeline?
Even this is not always accurately determinable! If you've had work done in your home, or heard anecdotes from others, you would realize this.
In those cases estimates form part of a whole basket of mechanisms and rules designed to ‘shield’ developers from users and vice versa, which imo is among the biggest mistakes one can make in such software companies.
It does seem like the responders have a little bit too much sunk cost in the whole famously fraught agile thing to understand critique of it unbiased. It's fine in theory but does have flaws.
That doesn't give you an accurate estimate, but it does usually give you a less inaccurate one.
And yet after three years at my first job, I could easily do a quick breakdown on a project, assign a three level complexity to each step (easy, medium, hard), put an amount of days corresponding to the selected complexity (to be reviewed after each project), add 25% for unforeseen events and contingency and my estimates were accurate enough most of the time.
It is not difficult to properly do rough estimates. Most developers just refuse to invest the time necessary to do it properly and sadly for some as the answers here already illustrate I do believe it’s because they think of themselves as special snowflake rather than team player.
The quality of the estimate would be unaffected.
The author made the claim that software engineering is fundamentally different from carpentry ("Almost everything a developer writes is unique, they have never built that particular thing before. We are not cabinet makers...")
But does that hold up? Many software projects do in fact have a similar analogue, and each house is unique and carpentry projects will thus run into unexpected snags.
The fact that the OP doesn’t see this is fact that they are too junior to be asked for estimates at all.
It really depends on what domain you are programming in.
I found that assumption to be ludicrous too.
Usually a business person makes some promise to the buyer or his/her boss that a certain feature will be done in X amount of time without consulting the team who does the work. The result might be overworking, resentments, stuff not being done in time and technical debt.
In the best case the feature will be done earlier, in the worst case you won't look like an idiot which promised things and didn't deliver and you won't compromise your relationship with the team. Compromising relationships with either clients or tech teams aren't good for your career.
More simply put, try to be as honest as possible. Not over promise, not under deliver.
One of my former girlfriends went to a business school and worked in banking. She thought people in IT have better salaries and she thinked tech people are a bunch of idiots and it's easy to push them to do things and extract a fat paycheck. So she did some courses on IT management and applied to lots of product owner and product manager role. After tens of interviews she hadn't had one success. To add insult to the injury, I showed her I make more money as a team lead than the product owner. Of course, the reverse is true and I've seen product owners being genuinely useful and having lots of business knowledge, some even programing knowledge.
Some people are just after the money, no matter what role they occupy in an IT company. And they tend to be the least useful persons who are blocking others to reach full potential.
Huh? Every major contract job on my house and my friends' houses that I have heard of has taken longer than originally "planned". More often than not unexpected expenses come up along the way. The same is true for commercial construction/remodeling. That's just how these things work; pretending otherwise is going to be a huge source of friction.
The real reason is to provide a scapegoat for the 50% of times it fails (ratio from Mythical Man Month)
"But but but but OUR organization isn't like that!"
Sure.
Relative estimates like planning poker can reveal hidden assumptions.
Commercial estimates can reveal risk appetite, desperation and more.
All estimates create insights but imho. it is best to burn the paper with the number after writing.
I imagine sometimes I renovate my kitchen, an expensive task, within my budget but with not a lot of leeway, I can't renovate the kitchen twice for example. If the workers say they don't know how long it will cost and they won't provide an estimate to when ... I would not be happy.
Even creatives, writers, musicians, graphic artists have deadlines and must estimate their work.
But we feel that our work is special and should not be asked to give an estimate.
I get all the reasons you provided, but at the end of the day an honest discussion needs to ask is estimating tasks effective.
Do developers get more done or less done when estimating tasks vs not estimating tasks? The only claim estimates would increase productivity is the extent they are deadlines. But I think you will see that when developers take meeting their estimates seriously their estimates tend to get more conservative. And then there can be a desire to not over deliver on the estimates to lend those estimates legitimacy. Without estimates, you still have a performance metric to use, individual velocity. If one developer isn't getting stuff done as fast as others, you don't need estimates to know they aren't performing as you would like.
Are estimates reliable? This of course varies greatly by the project and the team but I my experience estimates are typically no more precise than 50% of their length. So a 6 month project probably shouldnt take longer than 9 months. Your mileage may vary, but is there a way to crudely estimate a project without asking for estimates? I would say probably. Why not just measure the teams velocity and project it?
I would argue estimates were a first guess at a way to run a development team which has survived mostly because of a refusal to consider the first guess might not have been the best guess.
"Software development is not carpentry. Almost everything a developer writes is unique, they have never built that particular thing before. We are not cabinet makers repeating a variation of something we've built hundreds of times before."
Some software development fits this description. Quite a lot of it does not.
Having said that, most developers refuse to respond intelligently to these questions. The answer to almost all of these question should be something like "I think delivering this goal might take X resources, but here's the smaller goals we're going to deliver along the way to the big goal so that we can reposition when we go off-track."
Instead, developers say "Well we have to make sure that EVERYTHING can fit into this timebox, and our manager is going to treat this estimate as a deadline, so estimates suck!"
This question is supposed to be a conversation starter. Don't shoot the manager for doing their job. And even when the manager does a poor job and treats your estimates as deadlines, call it out! It's up to the developers to put up a bit of a fight and make more intelligent decisions too.
I've only met a handful of developers who are able to have a conversation like this. It's what the 'Agile' movement was attempting to fix. But it turns out conversations skills are tricky things to impart.
A few obvious reasons:
- The buyer has to make commitments externally, for instance to customers, partners, finance, marketing, his boss;
- The buyer has dependencies on those external resources and needs to plan for them;
- The buyer has a limited pool of resources and needs to know when they are free for the next task or project;
- The buyer needs to get an idea of costs to complete the feature and secure the budget;
- The buyer needs to make priority calls. If feature X is significantly more effort than feature Y then we can prioritise accordingly;
- The buyer is paying and simply wants to know when he will get his shit.
I wonder if the person asking the question would be happy to let someone do a job in his home with an uncapped budget and timeline?