This is actually one of the reasons I never hit the crossing signal in walkable zones: I feel it's rude when I could simply pay a little more attention and cross quickly without taking up others' time.

However, there are lights that do not ever open up to pedestrian crossing unless you hit the button, which irritates me.