I agree that how easy something is to learn depends on what you're previously familiar with. English certainly helps. Also, if you're already comfortable with math then you'll be more comfortable with symbol manipulation.
But that's a separate question from how easy it is to use a language after you've already learned it. The studies to measure these questions would need to be designed differently.
On the other hand, these questions do overlap. People who know a language don't stop learning; they get better at using it with practice. Also, people who use a language occasionally need to switch gears going back to it and need to relearn things.
Japanese does not have key words that are already in English. This is why it's generally considered harder for an English speaking person to read Japanese than to read some simple code.
There are orders of magnitude more people who will program in future than which program now. We should absolutely design things around those people.
Edit for natedad: 100% of people that fixed a bug were non-programmers at some point.