There are other categories of visas where you can receive the National Pension as well. I'm on a spousal visa and I'm pretty sure I can receive a pension even if I don't get permanent residence (or at least that's what they told me at the city hall). As glandium says I can also receive what I put in through the pension system in Canada and the UK (the only 2 places I checked). Also, if you don't want to receive a pension, you can also receive most of what you paid into the system when you leave the country. Unless you have only staid for a year or so, though, it's probably better to leave it there, financially. Finally, the city hall also told me that if you have been paying into a compatible pension system in another country, the number of years paying into that system counts for the number of years in the Japanese system (but you have to roll the 2 pensions together).

It's worth checking diligently to make sure that you don't fall through some bureaucratic cracks in the system, but it seems surprisingly robust.