Now silent films aren't a problem, as the text is shown before / after a given scene. So there is no processing two things at the same time. And it isn't a reading problem in general, because I'm an avid book reader and can consume books quite fast (although I only enjoy them if I read them slower and contemplate each scene).
I can only say that I absolutely prefer subtitles 100% of the time because the original performance is usually better. Good voice actors are not cheap, besides the fact that the sounds don't match the lips when it's a different language speaking.
I figure most people could probably do it with enough practice, though.
But I can't imagine watching, say, a foreign live-action sketch comedy show subtitled. I'd have exactly the bad experience you're describing.
If anything, they are usually at the other end of the spectrum, in that most local releases don't even include japanese subtitles (To be frank, I'm quite surprised this is even legal, considering the impact on deaf people)
So more than for artistic reasons, I think it's more often in order to lower the number of quality key frames required, so budget reasons.
I also can think of several shows that achieved significant fanbase in contexts where keeping an eye off the subtitles can be hard, while still being appreciated for their visuals: Tatami Galaxy, Monogatari Series...
"Screen-original" films and TV series are written in more a "top-down" style, with scenes being planned for their pacing, and the dialog in a scene liable to get cut down or rewritten to keep said pacing.
"Adapted from a non-visual source medium" films and TV series also almost always get their scenes cut down, just by necessity (a novel won't fit in 1.5hrs.) Though only "almost", because sometimes a director sets out to adapt a work into a really long treatment that can fit all the source material and then some — like the Lord of the Rings movies, or Game of Thrones. But even then, the director is still someone who is immersed in the top-down, "we get to knit it together how we want" zeitgeist of regular Hollywood film&TV production.
"Adapted from a visual source medium" films and TV series, though, are — if not heavily budgeted for adaptation polish — written mostly "bottom-up", starting with a treatment and storyboard that keep every line and shot from the source, and then letting pacing slip to keep in all the stuff that existing fans of the source material expect to see (and because the producer wants as many episodes as possible, so why not stretch one chapter out into five episodes if you can?)
This can lead to some extremely long shots, where there was dialog in a comic-book panel that was clearly not originally written with the intent of having it performed while holding/panning on the exact panel shown; but where a lazy "reading" of the source by the cinematographer leads to exactly that.
You can also call this the "make a radio play / audio drama out of the book; lock it down; and then add visuals to it" adaptation strategy.
(This is also common, as it happens, to specific genres that started out in the radio era. True crime, whether it be on TV or on YouTube, mostly consists of podcasts with the visuals almost being an afterthought. This is why so many people put it on in the background: the way it's done, you don't have to look at it to follow what's going on.)
The only downside is you have to actually _watch_ the show rather that staring at your phone most of the time...
You have multiple brothers with one girlfriend? I understand this is not likely, but it's such an interesting concept I kind of want it to be true.
I mean there's some undeniable fact that the original text is the original thing, and low-quality translations often just remove entire parts of sentences or details just because it's "annoying", and you are missing out on lots of fun little details. Not super relevant in paint-by-numbers TV shows but for shows with a lot of flourish in the dialog...
However, moving pictures will always be an artform that are heavily reliant upon the picture part. With subtitles, I lose my ability to focus on minor but important details in film. During dialogue, this includes subtle facial expressions. I also pay less attention to all other visuals, which I find difficult to do without subtitles off entirely. I feel like I miss out on a lot of potential beauty in cinematography with subtitles enabled. (Although, to be fair, there is usually less dialogue in parts that emphasize on cinematography / setting.)
Anyway, just throwing in my two cents. I used to think people who disliked subtitles were expressing some broader dislike of foreign films, so I found their opinion perplexing or even stupid. But I've come to that side of the aisle myself, not because I dislike foreign films, but because I realize that reading a small bar of text at the bottom of the screen will always handicap my ability to take in the original experience as intended by the director of the film (in 99.9% of cases).
I have tested out services that translate your "own voice" to another language and it was pretty cool, although I couldn't tell if the translation was quality. Automated translation isn't a solved problem, and this seems more of an issue than the tone of voice.