This is not true. If you have permanent residence you can retire in Japan and claim the National Pension if you paid in for enough years. In fact, recently the number of years of payment required was reduced from 25 to 10.
There are other categories of visas where you can receive the National Pension as well. I'm on a spousal visa and I'm pretty sure I can receive a pension even if I don't get permanent residence (or at least that's what they told me at the city hall). As glandium says I can also receive what I put in through the pension system in Canada and the UK (the only 2 places I checked). Also, if you don't want to receive a pension, you can also receive most of what you paid into the system when you leave the country. Unless you have only staid for a year or so, though, it's probably better to leave it there, financially. Finally, the city hall also told me that if you have been paying into a compatible pension system in another country, the number of years paying into that system counts for the number of years in the Japanese system (but you have to roll the 2 pensions together).
It's worth checking diligently to make sure that you don't fall through some bureaucratic cracks in the system, but it seems surprisingly robust.
Indeed, even though you have to pay all taxes like a local when you live in Japan, you can never benefit from stuff like pensions if you ever want to retire there.