Same issue, although exacerbated by a misspent youth near noise sources like teletypes and machine rooms. Certain high frequencies in my hearing spectrum were damaged which, as an audiologist demonstrated for me once, means I can't hear some allophones, especially from people with high voices (children, women). When its nominally quiet, my brain can fill in what those allophones would have been and I "hear" them normally, but when there is wide frequency noise in that spectrum the missing allophones cause the phonemes to become impossible to decipher. She had this tool that would play back words and could remove frequencies with a notch filter and replace the missing sound with either silence (still understandable) or noise (not understandable). Tuning a hearing aid to boost those weak areas works well for me with the exception that hearing aids don't benefit from the brains ability to focus on sounds in particular directions so they enhance the spectrum areas from all around rather than just where I'm looking.