I'm an Indian of American decent who knows Japanese and this has been my experience as well. My SO looks Japanese, but isn't ethnically Japanese, yet she gets immediately talked to in Japanese whereas I get the guest routine. I don't mind, and a simple answer in Japanese snaps them out of it, but it is tiring. Being brown skinned probably doesn't help.

I've thought about bringing my parents over but Japan just doesn't do cultural diversity, and I'm not sure my parents are ready for being Japanese in Japan.