I think it's simpler than that: something about the Latin languages makes them unusually hard to pick up for fluent English speakers. This strikes me as one of the main reasons Latin stayed being taught in the English speaking world for so long. Once you've got one the rest seem to follow, but that first one seems to be brutal.

Compare that with the Asian languages, and I know more people that claim to have picked up substantial bits of Japanese through importing videogames than is remotely reasonable, and have had a sort of light experience of it myself. The very alienness of it prevents attempts to link it back to your previous context, when constantly trying to do so is, in my experience, a perpetual tarpit.