I can remember when we watched television in class there would be no subtitles. The teacher would pause the video every 2 minutes or so and then ask someone in the class to explain what had just happened during those 2 minutes.
Usually it would be stuff like a news broadcast or similar the teacher had taped from Japanese tv broadcast.
You could mostly translate things because they were talking about relatively mundane concepts which were similar to what we had been learning. "That clip was about the opening of a railway station in this prefecture etc."
I can remember one lesson the teacher played Mononoke Hime in class, this was a pretty big anime movie which had just come out at the time it was full of references to Japanese mythology and folklore, no one is class had any idea what was going on in the story because it was full of language we'd never heard before.
The other thing is that anime tends to use a lot of informal language, which differed a lot to the more formal speech we learnt in class. So I suspect it would be very difficult to learn Japanese from Anime alone.
Grammar and concepts are comparable. It's no Chinese for sure. Much less alien than Korean too.
(And let's not try comparing it with Hebrew, Indic languages or Arabic vs Latin group.)
Personally, I don't think grammar is a large hurdle to learning to understand a new language. You can learn once that Japanese usually has the verb at the end of the sentence, and afterwards use a dictionary whenever you don't understand the word at the end of a sentence to learn a new verb. But using a dictionary is already a step up from passive consumption. Most people just want to enjoy the show, so they're not learning much.
Knowing grammar helps. I'm Italian and Spanish and French have such similar grammars that learning to read them was mostly a matter of building a vocabulary. Listening to French is more complicate as its hard to find spaces between words. Talking is both simple (similar sounds except a few ones) and complex (subtle differences in how to build sentences.) Writing is the most difficult skill. Everything has to be almost perfect.
English wasn't only a matter of vocabulary. The differences in grammar are large enough that building and understanding a sentence requires to think in a really different way. I got a shot at German a couple of times. The differences are an order of magnitude larger and I never had the time to go beyond the basics.
This sounds wrong to me…
(Though i think the correct pseudo translation would be “In English also verb can end go”. Subtle difference, but more like German - und auf den Punkt.)