I agree no estimation process is perfect. Nor do I think you should do comprehensive estimates on new work. However, the thing you want to know it's how much work there is. Not necessarily when you'd like to release a year from now.
Odds are high you have a deadline. So the incentive is high to keep the estimate below that line.
That aside, the incentive is high to keep the amount of work planned below the amount of work that can be done before the deadline. It doesn't matter how many abstract units or concepts you use, that's what you want to know if you have a deadline.
That is, if you give me a high confidence and a low confidence estimate, how do I know why you missed it when the time passes? More, how "close" to making it were you? If you got half of the work you estimated done by that time, I'd know you were about halfway there. If the date just passed, all I know is that you didn't make it.
And I used "I" up there. But it is really even more personal. All you know is that you didn't make it. You literally don't have anything to learn there.
Contrasted with any work you do. Look back and quantify what you did. Not how long it took to do it.
And only 90% of the work remaining!
It sounds like you are imagining an estimate as a standalone number that isn't revisited or given more detail. If that's how you do it then of course you can't learn from it. It's supposed to be an ongoing process - if you said '3-6 months' when we started then after 2 months I'd expect a different answer. I'd also expect the initial estimate to be accompanied by an explanation that talks about the work to be done and which areas are causing uncertainty.
If you know of anything to read, watch, or work through to learn to estimate tasks, I would be extremely grateful if you could link or describe it here.
As it stands, I’ve only ever been able to estimate tasks if I’ve done similar ones a few times before. If asked about a new type of task or one with a new toolset, I would currently have to refuse an estimate more fine-grained than a week—-the stress and shame of lying to you would be too much otherwise
Or would you feel more comfortable asking how much they will have to actually do? For example, they would have to disconnect the fridge and move it out of the way, disconnect the oven and move it out of the way, then buy enough tile and mortar for the space, which would be X boxes, and then clean and do the work.
Benefit of this way is when you come back, if none of those tasks has been done on day two, you know it isn't likely to get done on day two.
So, try and use the same approach for programming. Don't just say "it will take 2 weeks." Instead, say, it requires updating X component, modifying Y tests, incorporating Z new dependencies, etc...
As a team, you can try and portion size estimates on each of these. But don't spend too much time on that. Experience is the secret, not perfect estimates. (That is, the more things the team has done, the better they will estimate what they can do.)
I’m just worried that at some point, somebody will come along and ask for an estimate and I’ll say I cannot give one and that this means I am not really a professional.
Can't say if that helps strengthen your claim to professional, but that practice certainly sounds professional to me.
If it can be that far off (or more), what is the business value? And how do I know that the person I'm talking to will realize when I say "this will be done by the 20th" that I am thinking "I have no clue."?