Since Japanese culture/society will not accept anybody into their "inner circle" those immigrants will build parallel societies. A diaspora usually heavily connected to their original homeland and not their host. If those get really big, they will have influence in politics which again will change the face and culture of Japan forever.
This has nothing to do with racial purity. It has to do with what is in the best interest of the Japanese people.
Japan is more than a piece of land, or a group of people who make television programs and video games for you to enjoy. They are many millions of families each with a rich history that the world would be poorer for losing. And yes, if they stop having children or they lose their island then all of that will be lost.
Denying them this is tantamount to denying their humanity.
Immigrants (and furthermore children of immigrants) can and do respect and uphold the culture of the nation they move to. The idea that immigration destroys culture is patently false and there are countless counter-examples.
This view of immigration as "replacement" is just obviously wrong.
Uh, what?
After a generation, those immigrants will have children who are Japanese. Their native home will be Japan.
If your point is about ethnicity, then I don't see what the problem is. It's also ridiculous to suggest that Japanese ethnicity will cease to exist. Sure, you'd see more race mixing, but again, how is that a problem?
The US still has native American ethnicity, and that's pretty much as extreme as immigration gets - any modern immigration system is never going to have that level of disparity.
The quality of life for young people is not good in Japanese society right now, sounds like your motivation is more about preserving some notion of racial "purity" than anything that actually matters.