1) Developers cannot reliably estimate software timelines. That often comes with experience and age, but many never learn that skill. It is a skill like programming itself. It comes with practice.

2) Break down in communication between management (product managers) and engineers who are writing code.

On the first part. Usually junior software developers simply have not had experience shipping software in a full software cycle. From initial requirements, to design, to deployment, to maintenance. They did projects perhaps in school and homework exercises. Later some eventually learn that they chronically underestimate the time, and just start adding a factor (2x) to what their initial estimate is. That is a step up. But many never learn. There is also a race to the bottom -- the ones that promise shorter timelines get promoted and seen as smarter, more aggressive, better team players. The ones that are realistic are seen as maybe "taking it easy", "not working hard enough". Then everyone sees who gets promoted and show doesn't and learn -- "Aha I need to be like cool cowboy coder Joe over there, he is now running his own department". And so it spreads. Notice how there is often no feedback loop to punish the cool cowboy coder after the project is not delivered on time.

On the second part -- even if there are engineers who know how to estimate the time, nobody asks them. Because the person who promised the feature already played golf with the customer and told them they'll get the new shiny toy in 6 months. That person might have never been a developer, they might have an MBA or where a football coach or something like that. And they usually have a higher position in the company. So they promise and then everyone is pushed to deliver on that deadline. If it fails the failure is usually attributed to the engineers doing the work because they haven't worked "hard enough".

This is true. But this is the problem that Agile is supposed to help solve -- with the caveat that most companies don't do true Agile. I guess people are now calling it Continuous Delivery, but the idea is the same: you don't tell people when a feature will be done, only when you will be working on it. But even then, even with experienced estimators, the rule of thumb is to add 15-30% to their estimates.

> 2) Break down in communication between management (product managers) and engineers who are writing code.

Totally agree that this is very common. Unfortunately, the best way to solve this is with process, and developers hate process. But developers often don't realize that they can use the process to their advantage: a strong engineering lead can refuse to accept poorly defined requirements. That allows you to push back on product to force them to define the requirement better.

IMO this breakdown is caused by just that: developers who say "oh yeah, I can do that", then start working on the problem and realize that there are a lot of undefined edge cases. If they promised it to the client in 6 months, fine, but define what "it" is.

From the perspective of an MBA, if we have to bring in a contractor to implement it, fine. I know where I can go to hire certified rockstar contract developers at $200/hr, which I will do if the client is lucrative enough. There is no feature I can think of that would take more than 6 months of development.

In a top-down management style, your so-called "Lead Developer" is just the first head on the chopping block. They're leaders of puppet regimes. If the real leaders don't like them, they install a new one. It comes in many forms: layoffs, "performance" related dismissal ("you're not getting enough done!" because of all the rejected requirements docs), even reassignment to "re-engineering initiatives" that have no real chance of seeing the light of day.

Finding a management infrastructure that actually cares about feedback is incredibly rare. I have personally never found it, in 10 years across 6 companies. No amount of process is going to change a bad manager.

You talk about a business like it's supposed to be a democracy. It's not. You need clear direction, and democracy is messy and opaque if your company is too big for you to know the name of everyone who works there. Now the way you describe top down management is a bad implementation of it -- executives need to work to develop their middle managers -- but every company eventually has to convert from a democratic decision making process to a chain of command.

I've been working recently with a number of household-name startups that are having this exact issue. They've hit a plateau in their growth because the culture is dead-set against top-down management, but the company is now too big for people to self-organize. Engineering initiatives that would have taken 2 months in startup mode now take 2 years because they have to go back and rebuild half their systems to make them flexible enough to support the multiple use cases that they now have. Or because they were a startup, they didn't worry about geographic redundancy because that wasn't part of the MVP. So it was put off for later, and now they're a billion dollar company operating out of a single datacenter because their databases and applications were built without multi-datacenter being a requirement. Product can't even reliably tell engineering what features to implement because nobody on the product side has enough authority to shout over the crowd.

So having an infrastructure that cares about feedback is good, but the infrastructure also needs to empower people to do something about said feedback.

For every SV-brand startup, there are dozens of small-time consultoware shops serving their local market. They're the kind of places that have been around for 20 years and still only have 100 employees each. They can't ever seem to grow beyond that size because the one-way communication flow from sales to development keeps them in massive technical debt.

I agree with your second assessment, but it's more of an issue with the consultoware business model than anything. Most consultoware shops start up thinking they're going to get hockey-stick growth, but it turns out every implementation is super-custom and it ends up as a consulting business. Many companies fail to realize this has happened, and end up operating like a software company when they should be operating like a professional services company with hourly billing. Under a professional services model, if a customer wants a feature, you can quote them a price for developing and implementing it. Tech debt gets dealt with because it becomes a bottleneck for revenue growth when you can't charge clients to add functionality.

Estimation (i.e. pulling numbers out of your posterier) is not necessary (or usually desirable). What you want to do break down the problems into roughly equal size pieces -- the more the better. Each piece will take a certain amount of time. You can measure this and record the average. This becomes the estimated amount of time for each piece. Obviously the estimate has an error with some kind of distribution (my sleep adled brain is unable to remember what a likely distribution for the amount of time something takes).

It doesn't really matter because the sum of all the pieces will give you an estimate for the overall completion time for all the pieces. This estimate will have a normal distribution.

If you write down a list of all the pieces in priority order, you can draw a line at one piece of functionality and determine when it is likely to be completed, along with error bars.

Of course you are unlikely to actually to do the things on your list because if you were able to correctly guess all your requirements before you started the project then you would be in fantasy land. So what you do is tell the stake holders that this is what you are going to do unless they tell you that they want to change their mind.

Of course when you start they will say, "We're not going to change our mind. We're 100% certain that this is right". To which you reply, "Excellent". Every 2 weeks or so, once you have done some work and can show it to the stake holders, you should also have a list of new ideas. You should explain these new ideas and say, "Do you want to change the list to add these new ideas". Whether they say "Yes" or "No", you should reply with, "Excellent" and carry on.

Often the stake holders will have an "emergency" and decide that they have new opportunities which change your list. To this you should say, "Excellent" and change the list.

You should release whenever your stake holders are happy to release.

In this way, you have given the best estimate you can to help the stake holders plan. Any changes to the plan will be made by the stake holders themselves at the earliest possible time. You don't get caught up trying to deliver at any particular date because the stake holders are aware that they are changing the deliverables. You may have to ask them where they want to draw the line and give them new estimates for completion every time they change something, just to reinforce this idea.

The really tricky bit is breaking the work down into small enough pieces. This is very, very difficult and requires a lot of experience. The sucess or failure of the above will very likely rest on the ability to do this, so expect to pay a lot of money for someone with a proven track record doing it. It is often worth hiring a consultant if you have nobody on staff with this experience.

Hope this helps.

Disagree. Developers can provide reliable low risk estimates that will most certainly be rejected or truncated by management.

The problem is not estimates, the problem is every manager fantasizing that their Captain Kirk strategy,

    "I'll have it fixed in 4 minutes, Captain"
    "Do it in 2."

> the problem is any manager

If your greater point is that some developers really are poor estimators, I agree, but all of them operate under pressure to err on the optimistic side.

1. Every manager does that

2. Every manager that does that is the problem

I'm specifically talking about projects which go like this:

1) A set of high-level business requirements are put together, and given to a tech lead to estimate.

2) The tech lead's estimate, based on the requirements as-given, is used to determine a budget for the project, and probably a schedule as well.

3) Once the budget and schedule are approved, the project commences.

4) As development begins detailed software requirements are worked out, which usually exposes holes in the high-level business requirements that need to be fleshed out as well.

5) The estimates are never updated, and any attempt to increase the budget due to the new requirements are seen as a sign of failure.

What I see happening in #4 is two kinds of scope creep. The first kind is the sort everyone is familiar with: a requirement was missed, a new feature needs to be added, and since this wasn't in the original budget the budget needs to be increased. It's easy to identify these new/changed requirements, easy to see the big chunk of time needed to implement them, and easy to make prioritization decisions about the new requirement vs the other original requirements, so that maybe something can be dropped in order to avoid increasing the budget.

But there's a second kind of scope creep, and it's insidious. It comes from turning high-level requirements into detailed requirements into implemented and tested code. Every decision a developer needs to make about how to implement a feature is an implicit detailed software requirement, and you just can't know how many of those decisions will be needed or what their impacts will be until you get into the details. The time needed for these all add up, and I consider them to be scope creep because they aren't part of the high-level requirements used for the estimate and budget. We try to add padding for them because we know they're going to happen, but if you add sufficient padding across-the-board for all of the detailed requirement decisions that might be needed your project will never get approved. So we pad insufficiently, hoping that some of the requirements will be easier to implement than we estimate to cover the ones that aren't, and we don't manage this kind of scope creep at all. It's invisible, because it's just adding a few minutes here, a few minutes there, until our estimates have been blown away and we're over budget and we can't really say why.

The two types of scope creep is a good observation, I'll have to remember that idea.

These are all real human struggles. I have them for sure. I just wish you find a place with self aware leadership that endeavors to keeps these emotions in check. Usually startups with leaders who have families are a little better because their family essentially forces them to focus on someone else, at least a little.

3) Developers are expensive. If you have a team of 5 developers each being paid a salary+benefits of $1,500-$3,000 per week and they tell you a feature will take four weeks, management does a little mental math and says holy shit, that's a lot of money and it's understandably tempting for them to want to see if that team really needs four weeks, or if maybe it can be done in three weeks. (Meanwhile, as you said, the feature will probably actually take six weeks because developers cannot reliably estimate...)

4) Developers have a hard time saying "no." We have a culture of working crazy hours. Management knows this and often intentionally or unintentionally exploits it.

I am probably naive, but I think if we say: the situation is now X and I want to be Y, we can find a way to get there.

This works in our context (and has worked for many decades apparently). But I doubt this scales to a large size.

(Unless you work at Palantir, in which case I feel silly).

Of course that was just one variable and it wasn't isolated. There could have been other factors and reasons for the said result and it might not have had anything to do with "no MBAs" rules.

I just presented it as a reference or what others do as a "solution".

I suppose it depends on the business model the software is part of. If you want a pace that is compatible with proper engineering I would suggest looking into companies that deal with products whose lifetime is measured in decades such as CAD vendors or other highly technical software.

All freshmen to software engineering have this weird idea that a piece of code can ship once it builds and just shipping as much code as much as possible is the best road to maximize productivity.

Until they get hit in their faces time and again by recurring bugs and they start to understand that the way to manage long term software profitably is by making sure as few bugs as possible leave the software engineers desk, at which point best practices and long term planning starts to take hold.

That's at least my view, YMMV to a large extent, probably.

Just like juggling is a matter of throwing as many balls in the air as fast as possible ("What do you mean they come down again?")