I also grew up with subtitles, and I agree they're the best compromise when you don't understand the dialog. But I never acquired the skill to ignore them once I understood what was being subtitled at a native level, and I really don't really understand how anyone could.
The human vision is drawn to quick changes & movement, when you have an otherwise immobile scene that's mostly just dialog subtitles will pop in & out, I can't help but focus on them just like I would if something else suddenly popped into the scene.
If you have words pop into your vision you also can't help but read them, at least I can't, subconsciously reading the translation and hearing the original version at the same time is really distracting.
I found the only way to deal with them was to position myself behind some sofa so I could only watch the top 3/4 of the screen, but of course you miss a big part of the movie that way.
For me it's dubbing that's absolutely awful, because I want to hear the real voices of the actors, plus I learned English and a little Italian and French by watching movies with subtitles. I don't understand countries where dubbing is the norm, I hope it never catches on in my country.