If a million people were shuffling along a bridge and they all walked gently then the bridge will be damaged minimally, but if a hundred trucks that weigh several tons drive across the bridge then that damage would be quite severe. A million people regularly shuffling every day would be quite damaging over a long period of time, but the effect of a thousand large trucks is far more damaging.
My point is, listening to really loud low frequency sounds isn't as damaging as listening to really loud high frequency sounds.
Now, my comparison is really poor and heavy trucks cause far more damage than people, but the comparison between 'low frequency' and 'high frequency' is apt -- high damage to roads comes from high weight concentrated on a small point of impact in the same way that high damage to hearing comes from high frequency sound.
Except that bridges aren't self-repairing organisms, so the analogy is flawed in that sense too.
Just like getting your vision checked, you should get your hearing checked periodically. Age plays a factor. As you get older, the nerves in your ear become less sensitive and the structures in the ear can become less responsive to sound.