I guess I just see it as part of doing my job. I wouldn't be able to do it effectively if I didn't do this.

I don't think it's ever been secret in companies that I work for that devs don't spend 100% of their time on assigned tasks. There are always extra things like customer support, ops, urgent requests, unexpected refactoring that come up and have to be dealt with. Project managers I've worked with have mostly cared about the estimate being as accurate as possible taking into account these overheads.

The places I've worked that required time estimates (the wrong way to do it) still built in an overhead factor of ~30-40%. I.e., recognized that a dev would only put in 4-5 hours of 'coding' a day, with the rest taken up by meetings and random miscellany.

'An inflated time estimate' - well there's your problem. If it's a point estimate, as it should be, it's quite easy to do. Your 'real' velocity may be 10 points, but your planned velocity is 7. That not only gives you a little buffer in the event of something unexpected (so you're less likely to miss a sprint commitment due to things your team controls), but in the event nothing takes that time, it's free time to work on what feels important.