Because now your guessing on minutiae and will inflate the hours to reduce pressure. Or spending enough time to solve the task "estimating" the work due to the need to derisk it. Thus alleviating the pressure of being accountable.

Complexity points has value since they can be used in aggregate to estimate what a team is capable off, assuming the estimations on average are consistent between cycles.