I once read an argument that went something like this: if you're actually going to go to the trouble to build something, its expected return to the business should be so large that how long it takes (within reason) shouldn't matter. Conversely, if a project is only economic to undertake if it can be done within a certain amount of time, it shouldn't be done at all.

I find that I often don't have a good feel for how long something is going to take until I'm about 1/4 the way through the project. If I absolutely had to produce accurate, high-confidence estimates, I would just do 1/4 of the work first, then estimate the rest.

As to why estimating is so hard, here's how I like to think about it. Tasks form a tree: each project has a number of steps, each of which itself breaks down into sub-steps, and so on. We tend to estimate by thinking about the tasks near the root of the tree; but the actual time required is proportional to the number of leaves -- tasks we can implement without further breakdown. Without planning the entire project out in detail it's hard to know what that number is. But to plan to that level of detail requires making many design decisions; and researching and making those decisions is much of the work of the project.

The bottom line is, good estimates are expensive to produce, and my experience aligns with yours: it's rarely worth the trouble.

Agreed. Certainly, estimates of cost shouldn't be more precise than estimates of value, because ROI is the number that matters. And business is a black enough art that estimates of value are very rarely more precise than a factor of 4x.

What I think people try to get from estimates is a sense of control. But I think there are better ways of giving people not just a sense of control, but actual control.

This of course does beg the question as to how you know when you're 1/4 of the way though the project!

I wholly agree with this, and feel that all project plans should be constructed to mitigate the very high risk of initial estimates.

Too many project processes treat estimates as commitments with no backup plan when they (inevitably) go awry. I think a better approach would be to design your system around the idea that all estimates are suspect until proven otherwise, which of course leads to iterative approaches etc.

However, you can't run a business without any idea when anything will ship. Estimation is still, in my view, a very necessary step, so long as there is significant incentive to get it right rather than get it fast, AND the business is receptive to a little ambiguity through the project life cycle. This is a very difficult balance to strike.

In my experience you can have certainty in project scope or time, but not both.

Your point about estimate risk is excellent.

I agree you can't run a business without any idea when anything will ship. But to get a good idea when things will ship, you don't need estimates.

For example, in the Lean world there's something I've heard called "Disneyland scheduling". You break the project down into lumps. You maintain a queue of the lumps. You measure the average time from position X in the queue to being released.

Now you have a reasonable early warning system for when things will ship. Which is generally much better than what people have with most projects, which is a fantasy date.