I used to be the kind of person who couldn't work without headphones. Nowadays, I usually don't wear them unless there's a lot of noise I need to drown out or I just want to work undisturbed.

People who do use headphones all the time don't bother me though. I typically just send them an IM that they can acknowledge my question/comment on my own time.