The only "you don't belong here" vibe is from the constant "microaggressions" from being a visible minority - people will see your face and assume you don't speak Japanese, can't use chopsticks, etc etc, but those don't come from malice but rather ignorance. Once I open my mouth and speak Japanese all that mostly goes away.
I'm married to a local, have plenty of friends who are married and own property or businesses, no problems of any sort. Get along well with neighbors, business partners, hobby circles.
The biggest barrier will be work and business culture, where Japanese will have strict requirements on cultural integration (can't be rude to a client or make them lose faith in your company) and have higher requirements on dedication to work (over health and family) than westerners and that's where I've seen all the friction.
edit: I want to clarify that this is the experience of a white westerner. I hear things are very different for people from other parts of Asia (SE Asia/China)
My immediate guess, based upon dealing with people not Japanese specifically: Japanese culture welcomes outsiders who respect the culture. But once you try to get deep into the fabric of the community (buy property, marry a local, etc.) there is a strong invisible wall of which you were not aware.
Most cultures seem to have different ideas of where and how boundaries are created to push away outsiders. The notoriously cold Germans are initially very cold, but once you're on the inside there is complete trust. Meanwhile, the flaky southern California "let's do lunch" thing is an endless series of weak onion-like translucent boundaries and you never can get to the core of things.