It's much more difficult to actually confirm that the cause of the delay is an external factor, rather than a reflection of the person doing the work since most likely the only person with a real understanding of exactly why something is difficult is the person doing the work. So there's a stigma attached that isn't present in other professions.
You can always do more research, but research is at the cost of getting things done. So you balance that, with the understanding that sometimes you will go into projects with too little research. And no matter how much research you do, you will occasionally get genuine unexpected findings while doing the work that cause you to stop and go back into research mode.
EDIT: I mean, just watch those DIY shows some. Almost every episode, there is something "unexpected". Foundation work, roof work, load bearing walls, pipes and vents in walls, electrical or plumbing needs updating, etc. And it rarely ever actually effects their timeline, because they've done it enough to model that something unexpected is probably going to happen.
For instance, risk assessment isn’t all that helpful if your deadlines are bumped up by weeks or months and the client or your coworkers or bosses have no concept of why the project “can’t just be done immediately” when you’ve planned for added weeks. Even worse when they don’t understand why they can’t suddemly throw new variables at you after all estimates have been completed like new, potentially breaking features. Last minute scope creep is a real problem in some non-technical fields because the whole software solution is often some “nerd magic” and there’s no desire to understand the problem or solution except where to click and can they have it all by the end of the week. Theoretically maybe it could be done, but there goes all window provided for unexpecteds.
Admittedly, this is probably more of an organizational flaw than directly associated with the subject at hand, but a very real tangent.
Why can't you hit a hole-in-one on every shot on a golf course? You know exactly where the ball is, where the hole is, how far it is, which way the wind is blowing. All you have to do is hit the ball so it goes in the hole. It's definitely possible, there have been hole-in-ones on every green, so why can't you do that for this shot? Are you an incompetent golf player? Are there super-competent golf players who can do this on every shot? Why not? This does not seem difficult to me (but then I've never hit a golf ball in my life). Can you explain why you can't do this, but other people have done it before?
Also, if you are really the only one to understand (if you are the only tech in a startup for example), then part of your job is actually to explain.
Problems hit by developers are imho the same kinds that you describe with rocks: something unexpected has happened, we didn't/couldn't see it coming because of this and this, we need to do/change this and this, ... or maybe the risk was already considered in the estimate.
One big difference I was thinking about is that in the software industry nobody dies if something goes wrong. In building a tunnel, or shipping or many other industries, if you don't consider the risks well enough and ahead, someone might die.
When digging a tunnel, there is a finite number of kinds of rock that you can hit. It's well within the realm of possibility that an experienced engineer is familiar with any of the rock they will encounter, even if they didn't expect to hit them when digging. This is generally not the case with software. Even as an experienced engineer you continually hit issues that you've never encountered before and don't know how to solve without spending a significant amount of time doing research or trying out different solutions.
The upper bounds of complexity are higher in software than construction. It's not narcissism, it's just the nature of that kind of work.
It's a wonder that it works so well in fact.
I think software is probably the most complex and difficult engineering humans undertake. If you think of all the ways that a program can crash, compute incorrect results, and generally go wrong -- it's amazing that it even works at all let alone that in practice it works so well.
And we don't even have a widely accepted body of knowledge, capital-P professional obligations, liability, etc. We constantly hire people with no formal training, give them no formal training, and make no expectations of their expertise. We hire them for lots of money and put them in front of important projects and we promote them up. It's almost unreal.
Talk to a civil engineer or a doctor about what it took them to enter their profession and the amount of work they have to put in to maintain their license. And they get paid less than most software engineers.
And yet it's still somewhat unbelievable. Digging tunnels requires a lot of expertise and domain knowledge but as others have mentioned: there are only so many rocks out there to encounter. Once you have a thorough survey there are scant few unexpected problems you might run into.
But computing? Well... unless there's a battle-tested library that solves the exact problem you have then you're basically solving the problem from scratch. And one mistake is all it takes. You can't just add an extra I-beam or use a stronger steel... there's no fudge-factor in discrete systems. The whole enterprise of computing is managing complexity -- we have whole fields dedicated to the study of complexity with plenty of unsolved problems!
I believe it is just the nature of the work and why it remains so infuriatingly difficult to predict outcomes.
My pet response when someone asks why estimating software is hard: you've lost your keys in your house. give me an estimate on how long you think it will take to find them. I know you're going to think that if the keys are in the usual place it will take less than a minute. But if they're not... If you check all the places you might've been in the last few hours, maybe 5 minutes. But if they're not there then you'll have to think about how to search your whole house which could take hours. And what if you still don't find them?
And on and on.
This is often the case in other engineering projects as well.
But there's often less of an incentive to "confirm" that this is the case because if you (claim to) have hired the best people, to have given them the best possible training and working conditions, and every support in doing their job, it rarely makes any sense to go to them and figure out if they're really late because the problem is hard, or because they're just slacking off.
The investigation itself is rarely meaningful. Even if they are slacking off, what exactly do you expect they'll say? Yeah boss, we're just slacking off?
Managers who second-guess perfectly competent teams this way are usually shown the door (if they ever made it to managing something).
Of course, this is not all on engineers either. I'm sure everyone has seen the project manager who takes estimates as commitments. Cultures that allow that to happen will see repressing of generally good estimates. (If you use modeling, you just boost your confidence interval appropriately. 6 months at 85% or 9 months at 95%? Pick which number you give based on your experience with the PM!)
There's also a tendency to leave out important information. I recently worked on a project where 1) more than half of the features were not in the requirements because the product manager assumed we could re-use existing functionality from a completely separate application (we couldn't); 2) half of the requirements from the new work were left out of requirements because they didn't bother to specify them; 3) a critical requirement regarding how data was to be aggregated in search results (which is the primary feature) was not included because the architect didn't bother to tell anybody about it except for a passing "hey, what if we did this instead" before going on vacation. This caused 2 weeks of re-work after we were done, that would have taken 2 days if we did it from the start.
Our engineering team reviews all requirements as a team before giving estimates and this STILL happened, because we couldn't know what was missing. Good estimates begin with good requirements, and usually the people responsible for those do not end up being accountable, it ALWAYS falls on engineering.
Also ...
Do we really think that every project is such a unique snowflake that we can't build models?
Well, yes. That's inherent to what we do. Every project IS a special snowflake. If it wasn't we'd just download the software and install it. Oddly enough, installing and configuring software that is pre-made is very predictable and such projects rarely overrun. The sort of projects we're talking about are where pre-existing software won't do and thus, by definition, what you need is something special.
Meanwhile even very ambitious tunnelling projects like the Gotthard Base Tunnel, if you watch documentaries about them, they only hit a very small number of really unexpected issues even over long periods of time. Tunnelling is an incredibly predictable business. You buy some TBMs, set them going and worst case, you hit unexpected rock types or have an unexpected malfunction. The technology is advanced but rock is the same as rock has always been. Once the project planning is over requirements don't change. The course you're on doesn't change. Workers are largely trainable and replaceable - you never find you're unexpectedly relying on some random worker somewhere deep in the org chart who is 50x more skilled than their coworkers and who just got poached by a company that pays 3x as much as you can. The users/customers have no input: they get what you deliver and a tunnel is such a simple thing talking about whether they like it or not is a nonsensical question - it's a tunnel.
That's about as accurate as saying that software projects are only unique in trivial ways because there's probably already a free library or plugin that can do whatever else you need. It's easy to assume your occupation is much harder than everyone else's but we rarely have deep knowledge of the challenges other people face day to day outside of our own line of work.
As a simplified example, the design of a road, its pavement layers and its foundation is determined by factors like traffic loading characteristics, service requirements (peak capacity), speed limits and visibility, ground conditions, climate and more (see Caltrans' Highway Design Manual at http://www.dot.ca.gov/design/manuals/hdm.html), many of which rely on professionals in entirely other fields.
To make what I hope is an apt analogy back to software, you are basically saying, "Oh, their job is easy. All they have to do is follow established design patterns!" But, of course, we also have design patterns... Why don't we say the same thing about ourselves? Because we know that that's complete bullshit; there's a lot of thought and wisdom that goes into actually building stuff around those patterns.
Fogbugz created a feature called Evidence-based Scheduling (http://help.fogcreek.com/7676/evidence-based-scheduling-ebs). It's probably the only real quantitative approach that I have seen while working on web development projects. I'm probably wrong to assume that more "technical" fields like aerospace do much, much more than this, but it was really interesting to use this with what was mostly CRUD-work.
EDIT: added link.
More like: Previous work shoes that this project should take between one week and eighteen months at a 95% confidence interval.
...there aren't many projects one can do 300 times to narrow that window. And asking for a date instead of a confidence interval is asking for an ambiguous estimate in the first place.
We are not there yet, but are moving in that direction. We weren't previously even collecting the metrics required to create models in the first place, so we didn't have any data.
Of course I don't have any scientific evidence for this claim. But there are lots of phenomena that exist in the world that don't yet have scientific support. I think that software his higher uncertainty because it is much less constrained than physical engineering. There are a relatively limited number of ways that one part of a physical system can impact other parts, and those are constrained by the laws of physics. Software systems have no such laws. It's often possible for any component to write to arbitrary locations in memory controlled by any other component. This is kind of like a rusty hinge on a 34th floor bathroom door causing a problem with a doorbell on the third floor. Those kinds of actions at afar just don't happen in the physical world and if they do, they happen via pretty well understood physical mechanisms. To make matters worse, the discipline of software engineering is less than a hundred years old as opposed to thousands of years for physical engineering.
It's not about feeling special. I think there is something fundamentally different about software.
• Never upgrade your tech stack.
• Always use people of exactly the same level of skill.
• Assume the project stops at implementing a static set of requirements given up front.
• Assume the requirements are correct.
• Assume there's no political dimension to the actual deployment and usage of the app, i.e. the project is just software and not a business change project.
In practice none of these things are ever true. The business doesn't want a CRUD app, it wants an automated process that it probably can't explain, will change half way through the implementation and some of the employees may resist deployment of the software. And people expect software development to get cheaper and faster over time, as with everything else in life, so you can't exactly ship a Visual Basic 6 app in 2018 because that's what you happen to know best so you can easily estimate work with it.
It's really easy to tell whether you're building something standard. If it is standard, you don't need to write software for it. Say you want a standard blog. You use Wordpress. Say you want a blog that's kind of like Wordpress but a little bit different. You need to hire a programmer to write a custom component.
Effectively, all the software we write is custom. Once something becomes standardized, you no longer "build" that software. If it is standard, you install and configure it.
This is the same in the real world. If you need a standard kitchen, you can estimate how long it will take you to go IKEA, buy the cabinets, bring them home, and install it. If you want something custom, it all depends on how custom. If you want cabinets that retreat into the wall, have glass fronts, bulletproof countertops, can adjust in height so that both your 5'2" wife and her 6'11" husband can use them, in the mobile trailer that needs to tolerate temps from a dry -40F up to humid 100F without discoloration and warping, good luck. Oh, and they need to look really nice and be easy to clean. Estimate that!
TL;DR: when hiring a programmer to develop a project, nobody wants standard IKEA.
In short, I think there are very clear and tangible things that give software estimates a meaningfully higher uncertainty than other engineering disciplines.
In excavation, you can see the dirt and rocks, and say "This terrain is different than we expected." and someone else can see the same terrain and say, "You are correct. Our assumptions were wrong."
In software, you can't show anyone that the situation is different than your assumptions, because the assumptions themselves are the material that you're working with. Building software is the process of turning assumptions into results.
You're going a bit far there. How about writing, design, art, etc...? There are plenty of creative industries out there.
> It’s not because we’re narcissists who need special labels. It’s because people keep running into difficulty trying to square the very definition of a project (something with “defined scope, schedule and resources”) with our real world experiences.
Unless someone has an example of a project that can be easily scheduled and scoped, I am going to go with us being narcissists.
Whether it is writing a screenplay, interior designing the living room, designing an aircraft, or building bridge, scope and schedule are always tricky. No software required.
When something has a fixed scope and can be dependably completed in a fixed amount of time, the appropriate English word is “task.”
The unique problems with software I think are:
a) some things are genuinely novel: how long will it take to make a self-driving car? Nobody really knows.
b) big business projects may have incoherent or incompatible requirements, but the interactions between these appear late in development and the effort to clear them up is huge and fundamentally sociological - it has to feed back into changing the business process.
A lot of us have the issue of perspective. We believe we have problems no one else has and fail to consider that others may have problems we can't perceive. Out of sight, out of mind.
Also, millennia is already a plural.
Usually the client doesn't call halfway through the project and tells you that the tunnel you're digging really needs to start 10 miles further up the valley...
Oh, poor thing, you feel so special by being a software engineer?
Have you never heard of construction projects taking many times the estimated time/budget?
Maybe try digging a tunnel through a mountain, where the kind of rocks and caveats you find on the way can make you overrun your estimates? Maybe try sailing a container ship through an ocean, where a storm that appears in the middle can make your straight trip take a week more because you must go around? And we are not talking yet about building a subway line under a city, dealing with legal challenges, ...
Wake up, this is called a project, this is called risk assessment and risk management. It's maybe not done in your projects, but you don't need a fancy name for it, it's what has been happening for millennias.