It's way more than visual. For example, Japanese plurals aren't evident and need to be understood from context. Another example is the word order, in Japanese, the verb goes at the end of the sentence. Add to it Hiragana and Katakana characters and it becomes quite difficult to learn.

You're listing differences between Japanese and English, but my sample of anime fans who don't understand Japanese also includes Chinese and Korean speakers, for whom writing system or verb-final word order, respectively, are shared features. Different grammar from English isn't unique to languages with non-Latin writing system either. Latin itself has much freer word order, enabled by a more complex case system than English.

Personally, I don't think grammar is a large hurdle to learning to understand a new language. You can learn once that Japanese usually has the verb at the end of the sentence, and afterwards use a dictionary whenever you don't understand the word at the end of a sentence to learn a new verb. But using a dictionary is already a step up from passive consumption. Most people just want to enjoy the show, so they're not learning much.

I like to watch anime in Japanese because of the sound, but I don't know any Japanese so I look at subtitles, usually in English. I don't expect to learn any language in this way.

Knowing grammar helps. I'm Italian and Spanish and French have such similar grammars that learning to read them was mostly a matter of building a vocabulary. Listening to French is more complicate as its hard to find spaces between words. Talking is both simple (similar sounds except a few ones) and complex (subtle differences in how to build sentences.) Writing is the most difficult skill. Everything has to be almost perfect.

English wasn't only a matter of vocabulary. The differences in grammar are large enough that building and understanding a sentence requires to think in a really different way. I got a shot at German a couple of times. The differences are an order of magnitude larger and I never had the time to go beyond the basics.

In English also verb end can go. Plurality--they can usually be disregarded, too.

Order care can being if understood like you use you don't any you about.

> In English also verb end can go.

This sounds wrong to me…

Point this is.

(Though i think the correct pseudo translation would be “In English also verb can end go”. Subtle difference, but more like German - und auf den Punkt.)

Chinese grammar is refreshingly simple. I'm currently learning Chinese and it was shocking to me how easy it is to sensibly express ideas with very basic grammar (and some really funny idiosyncratic phrasings like… dong bu dong).