> Either that or they are not qualified for the job yet (hence they need to learn something new, which can be unpredictable)

IMO this attitude is pretty prevalent, and extremely harmful to software engineers. Software frameworks, methodologies, and techniques change very fast [1], and if you think that someone who needs to learn something is unqualified for the job, you are going to have an inherently adversarial relation with the engineers on your team.

Learning is and should be a normal and expected part of the day-to-day job of a lot of software engineers today. I think this contributes a fair bit to the problem of estimation.

[1] In contrast, I haven't seen a ton of novel developments in project management since the early 200s, we are still beating the drum of "Do Agile Not Waterfall" almost 20 years later

Yeah, the status-quo for me for several years has been "I don't know" because most things I build end up needing to debug something I don't have an immediate answer for, or construct something new, or talk to a new team, or find a new lib, or change a new setting, or migrate to a new platform. If I'm repeating work I'd done many times before, that's something that should be made more flexible and automated so I don't need to do it any more. (these of course exist, but you're pretty easily replaceable if this is a super-majority of what you do daily)

So my estimates are pretty much always "if it's well-built, documented, and fits what I expect, a week or less. If not, up to 2 months or more, but I should know more in 2 weeks". Getting a more accurate estimate quite often means spending as much or more time investigating than it would take in the optimistic case, so usually the answer becomes "give it a shot and we'll decide again later". Sometimes I have good news, sometimes not.

---

We sit on top of millions of lines of code, changing faster than we can read much less understand. You cannot know it all, nor is it worthwhile to try in the vast majority of cases. Poking into new territory is common, and the ability to do so effectively is a super important skill.

Code organization (part of Architecture) matters a lot here. Especially in large code bases. Software is meant to be “soft” or pliable easily. To do that, one should be able to alter a module without having to understand the whole system. If a small task takes too long, there’s a problem with the software architecture or code organization.

I think GP's "millions of lines of code" references the code that runs all the systems outside of your particular org's architecture.

Got it. Better to stick to tried and tested “boring” stuff and actually code small functionality yourself than chasing a thousand line library for trivial stuff.

At work, someone used a 3rd party library to send stats to a server in json format and “standardized” on it. 100s of developers had to now learn this new library and it’s APIs and one bug caused everyone a bit of hassle. It probably made the original dev’s job just a little easy but made it a bit hard for everyone else. These things do happen. The point is to learn from this and make better decisions over time, not give up and say we can’t get better. Trying to provide an estimate forces you to think more carefully and become better over time.

A poor library choice or two is not even 1% of the systemic dependencies any system has if it's working at a reasonable level of abstraction.

> "not give up and say we can’t get better"

Not sure why you feel the need to state this straw man? That is not my opinion, nor do I think it is the GP's.

It was not a counter argument to anything per se. Just pointing out that even with episodes of failed estimates, there is reason to continue estimating.

Let me clarify: when I said “they are not qualified for the job” I meant they are not qualified to produce the estimate yet. They can always go back and learn what’s needed and come back with a more educated estimate (still an estimate) or they call it out as a risk item and address it ASAP. More often than not, this is because they are not ramped up (manager’s or team lead’s fault) - why else would you ask a complete newbie, regardless of seniority, to estimate?

> 99% of engineers are working on things that CAN actually be estimated fairly accurately. New CRUD APIs? New ETLs? New React component? New integration with a product that has a published API? New service? New message format? New protocol? Whatever. Just break it down enough until it’s very clear what it takes to get it done.

What I can accurately estimate is how long it will take me to get it done.

What I cannot estimate is time spent waiting for people to finish their stuff, the lies that ‘the API is done’, the going back and forth with the stakeholders when they either change their minds, or I learn that what they want is actually different from the user story.

Like, all those human related factors.

Let’s say the code base is 1M Loc written by 230 engineers over 20 years. No docs. The epic is to make the whole thing support internationalization. That’s hard to estimate.

That’s a more typical requirement. A system that is built in nice incremental steps eg add an api and can be for its life sounds like a dream.

Looks like tech debt to me.

It’s understandably a big project. But you can still start breaking it down, correct?

Break it down by services first. Then do estimates for say 2 services to get an idea of what it takes to estimate the whole thing. Part of the process is also to communicate the cost to business so that they can see if the feature is worth it or not. (Read “Shape Up” by 37signals on this. Free ebook but you get way more than what you pay).

Can’t comment too much without knowing your codebase but it is engineering. Most tasks are not supposed to be this hard. If they are, then it’s time to start paying back tech debt so you have things under control again.

Tech debt is something that the engineers left undone because of pressure to finish quicker. The example above is not that. It's the fundamental assumptions getting changed mid-project.

The big system was designed with the assumption of single-locale users. This let them move faster, and make the system faster. It could mean schema changes, business process changes, and making hot paths in the code slow in a way that users will not tolerate.

Each service might have it's own level of integration with the single-locale, accounting might be 10 weeks of work, UI 5 weeks, and data processing 1 week. If you pick the wrong system to estimate against, you can easily be off by a factor of 10.

Estimates are definitely doable, but take a lot of time and expertise to do well, not just programming skill, but deep knowledge of the affected systems. An engineer can only estimate accurately for the pieces that they fully understand, and they can only estimate how long it would take them to do it, as the things they have context for aren't very likely to overlap well with other team members.

A dev that's been around for 3 years is likely to estimate lower than someone who's in their first year for many systems. But that thing the new guy implemented from scratch and the 3 year dev hasn't ever touched? Changing that will take much less time for the new guy. (all of this is on average, some people are better at holding context than others)

The book you refer advocates a concept of 'apetite' rather than estimation: "Instead of asking how much time it will take to do some work, we ask: How much time do we want to spend?"

Yeah. The reason I mentioned it is to highlight that the business often needs to know what something will roughly cost and cannot just headfirst five into it. Estimation is a form of communication that addresses this.