>We hear far more noise from the neighborhood in general than the neighbors we share a wall with. Granted they might just be particularly quiet people, but there is some significant sound-deadening in that wall.

I also share that experience. Unless you are using loud tools or do anything that vibrates heavily while touching walls (sound can travel very far this way) I never hear anything and I'm in a 5 story apartment complex. It's all about having nice and thick sound isolating walls. The worst noise sources all come from the outside.