That said, the difference between Japanese and Latin languages does play into the "foreign-ness" I mentioned. My very bad high school/college French is enough to feel comfortable deciphering a lot of Spanish and Italian, both written and spoken. Needless to say, that skill didn't transfer to Japan.
Again though, I love Japan because you truly feel as if you're in another culture. When I step off the plane in most countries, the first impression is sameness: same brands, same clothes, same affectations and language cadence. Flying to Japan truly feels like travel; I imagine it similar to going anywhere 100 years ago, before globalization (modulo British colonialism, of course).
It would be a heck of an adventure to commit to a few years' stay and really explore the country. I'd just want to do it with a partner to share the adventure with me.
I'm very happy you've had a great experience. I'm sure you will carry this with you regardless of where life's journey takes you.
For sure, do come over for a long term stay (with your partner!) and enjoy Japan. Like all countries, Japan has its own set of problems, but beyond that, it's a wonderful country. :)
A similar thing has happened to me and it doesn't make me want to go back.
I sometimes run into that kind of behavior, especially when my first impression is my looks rather than my voice. But then I think of it like this: they probably have to deal with many foreign customers, "most" of whom cannot speak Japanese. Or they might not have been exposed to many foreigners that they know there are foreigners who can speak Japanese.
At the same time, Japanese society puts consideration for others at one of the higher priorities when interacting with people, and is seen everywhere and highly evaluated by many.
So it feels natural for the person to assume that I, as another one of those foreign people, most likely cannot speak Japanese. Also, they could try speaking to me in Japanese, but it'd be inconsiderate of them if I didn't know Japanese. So many a time the default, and very likely considerate route out is broken English to the best of their abilities.
I know this makes people like us who do know the language a bit bad, but then again, we're still a minority so I wouldn't read too much into it. :)
Of course, if the person in question knows you know Japanese, yet keeps talking to you in English, then it might be better to assume something is not right. :)
However it's only happened once in a year of living here and never in Tokyo, only a ruralish city, so I don't take it as an indictment on Japanese people. I'm sure there are plenty of bars in rural America or Australia (where I'm from) that would have been equally unfriendly to someone of Asian ethnicity. At the end of the day you just have to decide not to let it ruin your day that there are some bad eggs in the bunch, no matter which country it is.
*Caveat though that I've heard it is much worse here for people from China than it is for white foreigners such as myself. If it made up a significant portion of my experience it would be hard to live here comfortably.
Literally right for almost every country: France without knowing French, US without English, Taiwan with Chinese or Taiwanese, Egypt without Arabic... Japan is not special in that regard, save for the fact that most people here have quite a low level of English (which is also not very specific to Japan).
So I wholly agree that making the effort to learn the local language repays by itself for any medium or long stay.
Actually, I realized this after writing out the whole thing but that sorry for the long rant is kind of like a good manners thing in Japanese internets. If you look up "長文失礼しました。", which literally means "Excuse me for the long writeup", it's usually appended to say a comment in a blog, or sometimes at the end of a Tweet when the speaker felt what they said is a bit on the longer side.
I somehow ended up picking up that habit somewhere down the line it seems. :)
The biggest thing I've felt since moving here: whether or not you can speak and interact in Japanese makes a world of difference. Seriously. I have interacted with expats who have not studied any inkling of Japanese, nor have tried any kind of interactions apart from other expats, and it felt their life was so bland. For example, they'd struggle to order anything except at big chain restaurants that have English menus, so that small alleyway restaurant that serves super nice ramen? Or that washoku (Japanese food) restaurant that's famous for its signature dish in Japanese? Very hard.
And that's without all the intricacies of interacting with Japanese people (beyond asking for directions where people are very helpful). Boy, that's a rollercoaster ride. Sometimes you strike up conversations with people so easily because you can speak the language. Sometimes you get hurt because of differences in ways of thinking. Sometimes you try to put yourself in a Japanese person's shoes and try to understand what they're thinking. To me, it's been one of the most fun parts so far.
And I feel much more welcome. For example, the bento (packed lunch) shop run by a family where I buy a takeout from once a week? They know my name and can tell my voice when I call to order a bento. I ask them how's life and they do as well as we chitchat while visiting their shop. I leave the shop with both of us saying "Thanks as always!". These small things do make a difference when you're living thousands of miles away from your home country.
I'm not complaining that there should be more English stuff in general; I feel people are trying to improve the situation. Nor that I'm saying not knowing Japanese will make your life hell; it won't. But, it will be bland. Especially if you happen to end up at a local town in rural Japan. That has its own charm points but you'd miss out most of them.
TLDR: Speaking from experience, if you ever plan to stay long-term in Japan, learn Japanese. I'm sure you won't regret it.
(Sorry for the long rant.)