True. But on the other hand an experienced developer usually can give a reasonable estimate. An if he says: "about two days maybe a week" then that is you best estimate. Using story points is just another complication.

And points do not stop anyone from holding you to account, if that is what they want to do.