In the time I spent learning English -- which is still far from perfect, especially in spoken form -- I probably could have learned quantum physics instead. Except, I realistically couldn't have, because good textbooks on this topic do not exist in my language, I wouldn't be able to learn more online, etc.
Now imagine removing an equivalent amount of knowledge from your life -- that is the price you would have to pay for not being a native English speaker.
(The idea of Esperanto as everyone's second language is that instead of some people spending 10+ years learning the common language, and some lucky people spending 0, everyone would spend 2 years instead. Until people would hopefully realize that private languages are useless, and then it would be 0 for everyone.)
A lack of incentive is what stops most.
> we can travel to a large chunk of the world and have a pretty good chance of being understood. We can work in globally relevant jobs with no language barrier. And so on.
You described the benefit of knowing English, not being a native English speaker.