I remember someone telling an interesting story on this topic, which I think illustrates the issue very well. If you happen into a neighbourhood that has a sumo training centre (called a heya), you will often see a sign on the wall that says, "No foreigners". Discrimination is not illegal in Japan.
An acquaintance of mine (and a foreigner) was well hooked up with a particular heya and he asked some vising friends if they wanted to visit the heya and watch the sumo wrestlers train. When they arrived at the heya, his friends saw the sign and asked if it was really OK. He replied, "If you know how to properly make an appointment, then you are Japanese as far as they are concerned."
I've been here 10 years and I'm definitely accepted by my family and my neighbours. And if there are people who don't accept me, then I will be the last person to hear about it (because that's Japanese culture). I won't say there isn't racism, but it's really on par with every other country I've lived in (Canada, US and the UK).
The requirements for naturalizing as a Japanese citizen are actually quite reasonable both in paper and practice.
I find its not as much as being in a place where the people would never, ever accept you as one of their own. To them, you're something to be avoided and barely tolerated. This is why Westerners have a lot of trouble in China or Japan, but an American spending a few years in Germany or England or France isn't a big deal. The elephant in the room is a certain level of acceptable racism that hurts foreign workers. Some nations are better at this than others.