I like to watch anime in Japanese because of the sound, but I don't know any Japanese so I look at subtitles, usually in English. I don't expect to learn any language in this way.

Knowing grammar helps. I'm Italian and Spanish and French have such similar grammars that learning to read them was mostly a matter of building a vocabulary. Listening to French is more complicate as its hard to find spaces between words. Talking is both simple (similar sounds except a few ones) and complex (subtle differences in how to build sentences.) Writing is the most difficult skill. Everything has to be almost perfect.

English wasn't only a matter of vocabulary. The differences in grammar are large enough that building and understanding a sentence requires to think in a really different way. I got a shot at German a couple of times. The differences are an order of magnitude larger and I never had the time to go beyond the basics.