Good point, maybe its the sound processing areas. A greater question is whether areas of the brain atrophy at all due to less stimuli coming through a specific organ.

All evidence I've seen point to brain plasticity; that areas used for specific things get re-mapped to other similar functions. For example, Deaf individuals experience a strong inner voice that activates similar centers in the brain as actual voices.