leicapowerball
I found that not speaking Japanese was not that big of a deal especially if you live in central Tokyo or one of the suburbs that have a lot of expats. Most businesses that you would likely frequent will have english menus or speak enough broken english to get by. I had a few health problems while I was out there and there are several clinics that have native english support.
The thing I found the most difficult about Japan is making Japanese friends. You will always be an outsider and in general Japanese folks just have different expectations when it comes to "work" friends vs "real" friends. You might think you're super close to a colleague from work but in reality you're a total stranger. It's not uncommon to not even know if someone is married or not. On the other hand if you're okay with having mostly expat friends, the community is very welcoming.
The only thing I like better about where we live now is the much higher salary (even with cost of living) and the access to outdoor activities. You'd have to take a train at least an 1.5 hours away to get to larger green spaces in Tokyo. Arguably, this might be more convenient for those that don't like driving but getting several kids ready and babysitting them on a long train ride is a bit difficult.