But they are accountable to their stakeholders (clients, upper management, etc). They take your estimate and many others from other people in the team and work those into a delivery plan.
A good manager will know how to manage risks and remove blockers in a way that gives a developer the best chance to work within the estimate. A bad manager will usually have no plan and put all the blame on the developers if things go awry.
Eg: I've seen many scrum teams transforming story ponts into days or vice versa missing the actual purpose.
I’ve been in the exact same situation a year ago, where another manager got mad at me for a delay. It was caused by him approving his entire team to simultaneously go on 3-month holidays when the COVID travel ban ended. Apparently that was my fault for not estimating accurately.