I can tell you from (thousands of) first hand experiences watching non-naive speakers of Japanese for many years: It doesn't matter nearly as much as locals want you to think it matters. Sure, the homophone thing is real, but Japanese people adapt their style of speaking depending upon the audience. (Japanese language and culture is highly context sensitive.) I hear it often when people pick and choose their words carefully in an attempt to reduce confusion around homophones. As a non-native speaker, when I am trying to use a relatively rare term that the speaker doesn't expect me to know, I slow down, use my hands a bit, and toss out some synonyms or brief explanation of the term I am trying to say. On the whole, Japanese people are excellent listeners, so it works pretty well.
What matters more: Japan has very little linguistic diversity for the size of its nation. Plus, it is an island. My theory (empirically observed): This makes them less able to adapt to non-native speakers. When you try to speak a type of Chinese (there are so many) to a native speaker from mainland China... their brain is automatically wired to heavy accents and different word choices, because their country is so linguistically diverse. As a result, when learning a Chinese language, it is pretty easy to speak with locals. In Japan: It is way harder. Mainland Chinese people really make an effort to understand you. It's no different than a tourist from a different region speaking to locals with a heavy accent ... or a different type of Chinese.
This all being said, after this interaction, I imagine you would have trouble in any country, with any language, because you seem quite insufferable and boorish.
Even after several years of learning Chinese I still had trouble communicating with Chinese people, especially those who had no experience talking to foreigners. When I arrived in China and asked the way to the university I was going to (which was close by and very famous) they just didn't get what I was saying. In the end I had to show them the written word.
I don't speak Japanese, but when I arrived and said the name of the city and they immediately understood where I wanted to go. After my experience with Chinese I was flabbergasted that that went so smooth.
I blame the tones in Chinese (which I admit I'm not very good at)
On the other hand, the Japanese pronunciation is one of the easiest in the world to learn, even taking into account the subtleties of pitch.