In most places "I don't know" doesn't have much effect. Just means the estimate comes from somewhere else. Another developer, or a PM.

People are just trying to make decisions about spending money, keeping clients informed, etc.

There are some research and moon shot type exceptions. But, for the most part, estimates are a necessary evil.