But it's exactly this 'ignorance' that is the major part of every software project. There are unknown unknowns, and there is no way from a standing start to grasp either their number or their nature. I'm happy if people are saying that this exploratory work should be done (in part or in total) before estimating, but in my experience that's just not palatable to most organisations.

I envy _hugely_ anyone who builds software that is even 50% the same as what they were working on six months ago.

Well, then let's call it what it is, because there are things that can be done to mitigate ignorance to some extent, if there is the will to do so from all parties.

> there are things that can be done to mitigate ignorance to some extent

Sure, but all you're doing is moving the uncertainty from the development to the planning phase, so instead of a development phase of unknown length and cost, you end up with a planning phase of unknown length and cost.

Firstly you you are still assuming the same gross exaggeration of novelty, and secondly, you can do analysis and planning at a much higher level of abstraction than development.

Waterfall gave the impression that planning and analysis is costly by insisting it should be carried down to the lowest levels of abstraction, before doing anything else.

Just to give an example of two projects I worked on at the same company within a few months of each other:

The first involved taking all sorts of situational data from various sources in an airport, e.g. radar systems with their own binary formats, and combining them with the work of a postgrad researcher who had come up with an algorithm to optimise when pilots should turn on an aircraft's engine in order to save fuel and cut emissions.

The second involved managing 3G dongles on fleets of laptops, tracking and sending alerts on their usage and billing info in realtime, back in the days when business users would accidentally run up £10,000 bills abroad.

If at any point in the first half of either of these projects I'd been asked to come up with an estimate for the total amount of work, it would have been wildly inaccurate. Each project had both low and high level surprises at every turn that would have just compounded to blow those estimates up. There is basically no significant way in which one informed the other, even though parts of the systems used similar event sourcing architectures on the exact same server-side stack, and were done for the exact same startup which broadly offered real-time systems to utility companies of various sorts.

It's entirely possible that there exists a breed of programmer who could have estimated these projects in their entirety, much earlier and more accurately than we did, but I've never met them, and I'd rather work in places that didn't compel people to.

I've known and worked at other places where basically any software methodology would (and did) work, but I certainly don't think it's rare (as you seem to suggest) to find yourself at the other end of the spectrum.

Aaargh20318 et al are making the claim that it is always inescapably like this and there's nothing you can do about it, because every day of programming is always fundamentally unlike anything you have ever done before. All I am suggesting is that this is an exaggeration.

Doing something about it takes some commitment, however. Nobody estimates well without having practiced it, and been honest with themselves about where it went wrong.

Understood, nothing to disagree with here.

> Firstly you you are still assuming the same gross exaggeration of novelty

I disagree. Even when you're doing mundane everyday stuff you have done a thousand times before, one trivial variation may throw of your estimate completely. Maybe an edge-case in this specific project triggers a bug in a 3rd party's library, and you're losing weeks waiting for the 3rd party to roll out the fix (been there, done that). You just don't know in advance

> you can do analysis and planning at a much higher level of abstraction than development

Of course, but then you'll get an inaccurate estimate.

> Waterfall gave the impression that planning and analysis is costly by insisting it should be carried down to the lowest levels of abstraction

That's only necessary if you want an accurate estimate. I think accurate estimates are pipe dreams.

There is a false dichotomy here between knowing everything and knowing nothing, between perfection and utter chaos. You may only have experienced the latter, but there are places where people know pretty much what they are doing, most of the time.

> ...there are places where people know pretty much what they are doing, most of the time.

It's called spec programming, where you have programmers slinging code against precise specifications. Common in ERP and regulatory or business compliance-driven code.

On the other hand, you have development. That is where you're attacking the less defined problems. Sure, we know the tools in our toolset, how to apply them, and time it takes to apply each tool. Problem is, the time to figure out the series of tools needed and their precise applications burns enormous time, time that could have been spent towards a MVP.

> Problem is, the time to figure out the series of tools needed and their precise applications burns enormous time.

In the use of the adjectives 'precise' and 'enormous', you are just repeating the same false dichotomy in a slightly different form, by ignoring (or not understanding) my point about the use of abstraction in analytical thinking.

The people who are most successful in what you distinguish as development understand a good deal more about what they are working on than spec programmers need to, and they are not, as you imply, groping around in the dark for a solution - that's what spec programmers do when faced with a development problem.

You aren’t wrong - for some things. I think it’s a spectrum.

For the things I work on, and frankly always have, I spend most of my time in the “R” part of R&D. Now granted, I’ve never worked on classical “business” or CRUD type problems.

It’s natural to think that once you have lots of experience, that estimating will be easier. In my experience, I found myself simply tackling harder or unique problems. Instead of getting comfortable, you find yourself mostly operating at the edge of your competence. This is great fun [1] but makes estimating pretty hard.

[1] I personally enjoy being on the “black diamond” part of the learning curve. (“Black diamond” = most difficult ski trail)

It's like building a deck except even the screw hasn't been invented yet. And you think: I'll just create a device that attaches the wood like a big paperclip but then realize after you make it that it's not strong enough and won't scale so you create a nail. But then you realize you may need to reuse the wood so you finally create a screw.that can be removed.

That analogy might have worked in the 1950s.

Agree 100%. IMHO, a lot of it has to do with the programmer mentality of wanting to be "challenged" to solve problems one step away from their competency. If you're fresh out of school, implementing your own allocator might be a challenge you take upon to solve a memory constraint, when you can just take what someone else (with more experience) has built. And then senior people insist on designing yet another over engineered platform technology because everything has to be 'future prof' (when in reality it never is and only ends up being performance-proof). How many JS frameworks have we had so far that do the same thing and just repeat mistakes of previous ones? Its like one every year.

I agree.

There is also a lot you can do to decrease impact of novelty on the overall schedule and most people I work with do exactly nothing to reduce variability in the process and improve their estimation ability.

It is easy to claim every time you write code you are doing something new but, hey, every time they build a building they do something new. Yet buildings (except for extremely ambitious projects) typically can be delivered reliably by reputable developers.