Yup, I wear my Bose noise-cancelling headphones on the subway most of the time without music -- it just takes the edge off the rumble, the noise, the screeching.

It works really well on the subway, since the frequencies tend to be lower (similar to planes). Outdoors it's less effective, since the bothersome noises are the higher-frequency sirens, the beep of backing up, jackhammers...