But you said it yourself: you need a visa. It's not simply a question of whether an American or anyone else finds it satisfying to live in another country. The government has to want/permit you to be there. And in the case of Japan in particular, that's a difficult situation to navigate, especially if you are looking for a place to live for your retirement years.