Sometimes we don't know what we don't know. If we don't know what we don't know now and if we don't know what we'll find we don't know later, then we don't know how long we will need until we'll know.
Uff. Sorry. If I had more time, I would have written a better comment.
If it's the latter case, then a good developer can describe the immediate steps they'll take and what questions they expect to answer. Whatever the case, there is always some concrete next step you can take to which you can assign a reasonable estimate of time. That might just be that you'll spend one day perusing some documentation, and the only outcome you expect from that is that you'll develop a reasonable enough understanding of how large/complex the documentation is, such that you can come back with an estimate of how long it will take to find and carefully read all the relevant sections of the documentation.
That's totally fine - if it's the sort of wildly unknown problem that you're describing, then I recognize we'll have to take steps to understand the problem. There's still never a case where it's impossible to create an immediate plan of action and understand roughly how long that will take (unless you are simply unqualified to take on the problem).
I had a boss loved the phrase, "a date for a date." He just meant that if you can't give me a date for when the thing will be done, then you need to give me a date for when you can give me a date for when the thing will be done. Can't do that? Date for a date for a date. However many levels we have to go, you're not getting out of whatever meeting we're in until you can give me a date by which you'll be able to come back and give me another date.
Right now I’m replatforming a legacy app. It’s undocumented, developed by dozens of staff — all of whom are gone, and full of crazy bugs and gotchas.
Out of the box it wouldn’t compile, the source code was missing files, etc…
The task is to find every such hidden issue, fix it, which then allows us to find the next hidden issue, and so on.
When we’re done finding issues, we’re basically done.
Can you tell me how many issues we’ll encounter? I can’t. I’m being perpetually surprised by the code base, finding random binary blobs that actually are from some other project we didn’t even know exists until that moment.
Fundamentally, it’ll take as long as it takes. The application is mandated by law, and it must be made to work. It’ll cost what it costs, and it’ll be finished when it’s done.
That's exactly it. Scoping out the solution is 90% as hard as just doing it. By the time you've figured out enough of the details that you're sure there are no more surprises waiting... you're just about done.
I suspect that a lot of counter-examples people have witnessed are examples of poorly factored codebases. I've seen teams/projects where they estimated projects accurately because the "new" thing being created was a slight variation on a kind of thing they'd created many times.
For example, I once worked for a company that generated e-commerce sites for clients. They'd make a new one basically by copy/pasting an existing site from another client and changing some text/images here and there. Super predictable after doing this a bunch of times... but also a maintenance nightmare having dozens of slightly different codebases. And how do you iterate? They couldn't evolve.
Shopify ate their lunch. I bet Shopfiy has a really hard time estimating the time to ship a new feature. But they add it once for everybody.
I love deadlines. I love the whooshing noise they make as they go by.
- Douglas Adams
A knowledgeable developer can predict how this cycle goes and explain that an estimate of the whole task is impossible, and they will be quite literally and completely correct in that case.
Ah, that would be nice. In my experience it's more like, "Okay, I understand it's not estimable at the moment. Just make your best guess and we can revise it later. (Psych! I will hold you to this estimate like you were under oath in court!)"
This leads to things like the Scotty rule. Since management will always misuse, misunderstand, or misrepresent your estimate anyway, might as well look out for number one and inflate the estimate, just in case.
The only time I've really had management "misuse, misunderstand, or misrepresent your estimate" is once they got past the point of being several hundred people with the bureaucracy that entails. Then I leave and go to a startup that's <50 people.
The correct solution here isn't just to insist estimation is impossible and be offended that someone would ask for it - that just misunderstands how businesses work.
The correct thing is for either of those parties to say, "Okay, I understand it's not estimable at the moment. What are the steps that we need to take in order to get a reasonable estimate?"
A good PM will do this, but where there is a bad PM, a developer can step in and say the same thing. In your scenario, "The developer says “there are too many unknowns to provide any sort of estimate”." What the developer could say is, "There are too many unknowns to provide any sort of estimate - let me give you a few examples. <examples>. What I can do is come back to you after <amount of time> with a list of the unknowns and an explanation of the work required to evaluate them, so that we have enough information to estimate. Once you have that list, then you can decide if we want to proceed with the work to understand those unknowns."
I totally understand why developers get frustrated when they're asked to give estimates for things that can't reasonably be estimated with the current information available. But the flip side is it's also very reasonable for PMs to get frustrated and ignore developers who insist that estimation is an impossibility, because that's virtually never true - maybe it's impossible to do now, but there is a path to make it possible. If you work proactively to expose that path to the PM, it'll generally result in everyone being happier.