Human change does not require Darwinian selection. Look across cultures. That people might tolerate or be most comfortable in noisier environments could vary for non-heritable reasons. Some people like less stimulation, some more. Autism rates have changed faster than our genetics also.