That which is passed off as novelty is often ignorance (not necessarily on the part of the developer.) To continue the retail analogy, software projects are often like the guy who is building a deck on his house, but keeps going back to the hardware store because he made no attempt to think things through, or measure before cutting (note that, in this analogy, the would-be builder is not necessarily playing the role of the developer. In some cases, that might be the hardware store owner's role.)
I envy _hugely_ anyone who builds software that is even 50% the same as what they were working on six months ago.
Sure, but all you're doing is moving the uncertainty from the development to the planning phase, so instead of a development phase of unknown length and cost, you end up with a planning phase of unknown length and cost.
Waterfall gave the impression that planning and analysis is costly by insisting it should be carried down to the lowest levels of abstraction, before doing anything else.
The first involved taking all sorts of situational data from various sources in an airport, e.g. radar systems with their own binary formats, and combining them with the work of a postgrad researcher who had come up with an algorithm to optimise when pilots should turn on an aircraft's engine in order to save fuel and cut emissions.
The second involved managing 3G dongles on fleets of laptops, tracking and sending alerts on their usage and billing info in realtime, back in the days when business users would accidentally run up £10,000 bills abroad.
If at any point in the first half of either of these projects I'd been asked to come up with an estimate for the total amount of work, it would have been wildly inaccurate. Each project had both low and high level surprises at every turn that would have just compounded to blow those estimates up. There is basically no significant way in which one informed the other, even though parts of the systems used similar event sourcing architectures on the exact same server-side stack, and were done for the exact same startup which broadly offered real-time systems to utility companies of various sorts.
It's entirely possible that there exists a breed of programmer who could have estimated these projects in their entirety, much earlier and more accurately than we did, but I've never met them, and I'd rather work in places that didn't compel people to.
I've known and worked at other places where basically any software methodology would (and did) work, but I certainly don't think it's rare (as you seem to suggest) to find yourself at the other end of the spectrum.
Doing something about it takes some commitment, however. Nobody estimates well without having practiced it, and been honest with themselves about where it went wrong.
I disagree. Even when you're doing mundane everyday stuff you have done a thousand times before, one trivial variation may throw of your estimate completely. Maybe an edge-case in this specific project triggers a bug in a 3rd party's library, and you're losing weeks waiting for the 3rd party to roll out the fix (been there, done that). You just don't know in advance
> you can do analysis and planning at a much higher level of abstraction than development
Of course, but then you'll get an inaccurate estimate.
> Waterfall gave the impression that planning and analysis is costly by insisting it should be carried down to the lowest levels of abstraction
That's only necessary if you want an accurate estimate. I think accurate estimates are pipe dreams.
It's called spec programming, where you have programmers slinging code against precise specifications. Common in ERP and regulatory or business compliance-driven code.
On the other hand, you have development. That is where you're attacking the less defined problems. Sure, we know the tools in our toolset, how to apply them, and time it takes to apply each tool. Problem is, the time to figure out the series of tools needed and their precise applications burns enormous time, time that could have been spent towards a MVP.
In the use of the adjectives 'precise' and 'enormous', you are just repeating the same false dichotomy in a slightly different form, by ignoring (or not understanding) my point about the use of abstraction in analytical thinking.
The people who are most successful in what you distinguish as development understand a good deal more about what they are working on than spec programmers need to, and they are not, as you imply, groping around in the dark for a solution - that's what spec programmers do when faced with a development problem.
For the things I work on, and frankly always have, I spend most of my time in the “R” part of R&D. Now granted, I’ve never worked on classical “business” or CRUD type problems.
It’s natural to think that once you have lots of experience, that estimating will be easier. In my experience, I found myself simply tackling harder or unique problems. Instead of getting comfortable, you find yourself mostly operating at the edge of your competence. This is great fun [1] but makes estimating pretty hard.
[1] I personally enjoy being on the “black diamond” part of the learning curve. (“Black diamond” = most difficult ski trail)
There is also a lot you can do to decrease impact of novelty on the overall schedule and most people I work with do exactly nothing to reduce variability in the process and improve their estimation ability.
It is easy to claim every time you write code you are doing something new but, hey, every time they build a building they do something new. Yet buildings (except for extremely ambitious projects) typically can be delivered reliably by reputable developers.
It works only very special circumstances: when the task can be very well defined because it has been done a lot of times before and the people doing the task are very experienced on it.
But the caveat is that if you are doing such a task you're not really doing any innovation at all. Actually, if it is so well defined and trivial it should probably already have been automated.
True innovation must have a lot of unknowns, it isn't innovation without it.
I’m kidding, of course. It really depends on how well business and development understand each other.
Yup. This is the crux of the whole issue, and it's being completely blown out of proportion. There's always going to be details the estimator (dev) won't know, but the more clear a picture you have in your head, the more confidence you can have in an estimation.
This is directly related to the size of the project too, of course. If it's a 12 month project, a single developer is massively unlikely to have a reasonable understanding of all the potential roadblocks that the project will take. However a 1 day task, with tools you know/etc, you can have a pretty good idea how long it'll take.
There's always some unknowns, but that's literally what an estimate is for - attempting to plan for the unknowns. I can write an add function and I'd bet a lot of money that it won't take more than a day.
Unreasonable management seems to be giving developers PTSD here. As I'm having trouble understanding why the idea of an estimate, by definition exactly what we're doing, somehow doesn't apply to software development.
If you're being pedantic you could say the same about a lot of things. How long to dig a 1foot hole in open ground? Well who knows, there might be a massive rock - but that doesn't mean you can't give an estimate. It's literally what an estimate is for, an approximation based on assumptions and expectations.
It's inexperience all around. Also, most places have no clear long term track, so people bounce after 2 years rather than growing to understand each other.
Managers are generally free to cut estimates in half, and move to a new role or move to a new organization when things fall apart. They're not trying to be villains, they have a lot of pressure coming from a lot of directions and think that estimates are bogus anyway, thanks to star trek and the occasional miracle that happens. (sometimes things really are much easier than they seem)
Developers are optimistic, wildly underestimate, and feel bad when they miss them. The overreact and start massively overestimating. When they progress to the next stage they take the time to give a good thorough estimate to have it slashed by a manager, who knows they overestimate.
Nobody sticks around long enough to really understand the state the other is in, so you get this vaguely Kafkaesque world that makes everyone a little crazy.
Software isn't like building a bridge; it's like planning to build the bridge.
I'm pretty good at software estimating; I can roughly tell when something will take an hour, 2 hours, 4 hours, a day, a week, or month, 3 months, or 6 months -- in those intervals. It's more difficult and time consuming to get more specific than that.
I've also had a 1 day task blow out to entire week more than a few times. At that point, you have to start questioning the predictive value of estimates.
If I estimate a month to build your feature and it takes me a year or two, you're not going to be very happy...
This is the reason big design up front hasn't been the gold standard in planning. Most modern methodologies acknowledge that there'll be a lot of changing requirements and learning-while-building.
The point is, given this information, you could a) run around like crazy doing all manner of reckless and crazy cowboy programming and putting out fires b) carefully plan a relatively short period of time on a small, achievable, well defined goal (not "build the project"), then making a note of what worked and what didn't, and readjust before you plan the next short period of time
You need to have some fairly specific goal in mind to start doing anything. AKA are we building a PS4 game or a self driving car?
Methodologies like Scrum are all about transitioning a long term objective into short term progress. But, you really do need some fairly well thought out goal.
Further course corrections quickly get very expensive. Sort of like sailing from Virginia to Europe then deciding 1/4 of the way in on France then changing your mind to Spain at the midway point is probably fine. However, changing your mind to Australia is a much bigger deal.
I appreciate that sometimes hard deadlines are necessary but if I can I'll always avoid them.
More charitably, they're an opportunity to introduce system 2 slow thinking on the team.
There are certainly novel engineering problems that simply won't yield to estimation, but there are also lots of well-solved problems that will. Managing the latter isn't desperate, its just sound practice toward reproducible results (ideally deployed only where reproducible results create value).
Certainly when you're writing a double-linked list a third time you can estimate how long will it take you. The problem is that in real life you never solve sufficiently the same well-solved problem. Feature details, matters of integration with other pieces of the system intervene - and in practice those errors introduced by new circumstances are significant comparing to predicted time. So the practice may be sound, but it doesn't give the desired results, like predictability of required time.
If you're looking for a formula that lets you estimate once and then move on to the "real work" now that that tedious BS is out of the way, there isn't one. But you can still do something. There's ground between "I can't estimate that in 30 minutes" and "I can't estimate it at all" and it's up to you and your stakeholders to come to terms on how to balance the risks in your estimates (in broad terms: too inaccurate estimates vs too much time wasted estimating requirements that have since changed).
(There's a parallel theme in some other comments: "if it's a well-understood problem why am I not using the existing solution?" Similarly, there's a difference between "well-understood" and "perfectly abstractable and automatable", otherwise almost none of us would be employed these days. :) )
But if it's a solved problem, why not use the existing solution ?
I think what's missing here is that I can predict how long something is going to take when I talk with the managers/feature-planners about what the feature is. If I feel I have a solid understanding of the requirements, which pieces of code it'll touch, etc - then I can give a reasonable estimate.
Now, the bigger the project the more difficult this becomes. Details I've missed is always a possibility, and that just stacks up the bigger the project is.
So I think being specific in this type of discussion is really important. A feature that's going to take 6 months to implement is insanely tough to estimate, and very likely to have a wrong estimate. A feature that takes a week can pretty reliably be accurate within a day or two, in my experience.
I'm not defending management in these sorts of things.. I think it's our job to not only give estimates, but include understanding about the expected validity of that estimate.
All estimates are worth while though, your post makes it sound like they're not. You tell me you need a feature that's going to take 12+ months? It's so hard to predict that it's laughably wrong.. but that's still insanely useful. Why? Because now the management knows it's highly unlikely to be a 1 week, 1 month, w/e feature. It's big.
I'd argue you can't.
Right now I'm working on a feature that I estimated as 'a couple of hours'. I've been working on it all day by now. Not because the feature is that difficult, or because I didn't get a set of good requirements. The requirements are crystal clear and should be easy to implement.
The reason it's taking so long is that I'm fighting with a UI component that doesn't behave as expected (or documented). What should have been a trivial part of this task, a matter of seconds and not worth estimating at all, turns out to take all day because I didn't know this specific component is misbehaving and I couldn't have known until I started implementing it.
You cannot predict things like that without doing it.
I've done this 50 times before (10 times in this project!) and these kinds of issues still occasionally occur. How does anyone predict for this kind of thing?
Just because you're wrong sometimes doesn't mean you can't be right more often than not.
Another way to look at it is, as a company how can you budget anything if you can't make even the slightest of estimations?
What you're suggesting results in throwing away any ability to budget, roadmap, coordinate, etcetc. Absolute prediction is of course impossible, but if what you're saying is true than I should buy a lottery ticket because my estimations are insanely lucky.
And sure, I get estimations that run long, we all do - I'm not saying otherwise.
But to clarify: You can't predict things well that you haven't done before.
The point being, when trying to predict an unknown it's best to give a range and not a fixed estimate. If possible.
That said, requirements emerge daily. Transparency is essential for honest dialogue.
I agree and disagree with the notion that an estimate can only be provided after the work is done. If the work is broken down properly, you can use empiricism to get a good understanding of what is needed based on previous, similar work completed. But to think an estimate can only be provided after the work itself is done is kind of ridiculous. If you section off variables with manageably-sized work you should have a good understanding of what's needed to pull it off.
Typically no one wants to pay more for less, but that's the best strategy to deliver high-quality software on time.
You reduce the scope drastically but do not adjust the timelines. You pay for more development resources(tooling, developers, servers, time, etc.).
If you can't deliver at that point, then there's a serious fundamental issue with either the leadership or the team or both.
Startups don't have this luxury. So they must execute on the least complex scope on a very tight timeline. High-complexity scope increases the fatality risk to the company.
So as the scope increases in complexity the risk increases and the likely the company itself will fail increases. Again, because startups do not have the cash to pay for the proper amount of resources.
I think that is a myth right up there with “If I was rich, I’d be happy.” Massively funded software projects fail all the time.
If you're really working on new problems all the time, consider yourself extremely lucky.
- how long will research take? Where do you buy milk? Where is the nearest shop? 3 mins.
- how long to get to nearest shop and back (eg 10mins x 2) = 20 mins.
- find milk in shop (30 secs to 5 minutes), so say 4 mins.
- Buy milk. (1 to 5) say 3 mins.
Don't forget that these are estimates and always add a bit of headroom. The above comes to 31 mins, so say 35 and you know you're covered. If the manager isn't happy with it, give him the low down figures.
If you can plan and execute putting a person on the moon, you can plan and execute software development.
I think the problem is the overwhelming majority of software development is so low stakes, easy to replace, assumed that bugs are par for the course, etc... that most orgs have stopped putting real effort into planning AND executing on that planning, instead just cargo culting the planning phase.
Sure once all that expensive and unknowable planning and development is done, you can double-click that executable and put a man on the moon.
But the budget for planning is limited. To realistically plan out most software projects would entail almost as much work as simply executing on them. That level of resources for planning vs. execution is acceptable for putting people on the moon. Probably not for building a line of business app.
If you can accept those kinds of preparations and also agree to not change the specs much (i.e. no, we aren't going to Mars instead) after we've started developing it I'll say we can get you a pretty nice estimate.
I think you vastly overestimate the quality of the software and systems that took men to the moon. Try reading https://www.google.com/amp/s/amp.space.com/26593-apollo-11-m... for example - you might conclude "they got lucky"
Do we have the planning and estimates of projects that are similar to "put people on the moon"? I'd like to see how the plan/estimates of these projects look like and how they change.
Okay, but I think you've missed the point. Planning is hard, and you can't possibly account for all the variables.
Everything after that statement I can agree with
To use the 'going to the supermarket for milk' example. I could make a fairly accurate estimate for that because I've gone to the supermarket hundreds of times. I know all the different things that can go wrong and account for them, because I've encountered them before. The elderly person who wants to pay cash and has a bag full of coins. The guy who finds a 10 cent discrepancy on his €92,30 receipt and has to argue for 5 minutes with the cashier (while the row behind him keeps growing). etc. etc.
Now imagine that today is your first time going to a supermarket. In fact, before today you had never heard of the concept of supermarkets, or milk for that matter. How good will your estimate be ?
The only time you can make a decent estimate is after you've finished. Or to put it differently: making an accurate estimate is possible, if you're going to accept that making the estimate is going to take a long time, but I can't tell you how long.
Development time estimation, and every methodology that attempts it (I'm looking at you, Scrum), are little more than desperate attempts by managers to feel in control and relevant.