I agree there is a risk that bad actors could exploit this vulnerability, but there is also a risk purging voters prevents legitimate voters. That is especially concerning when the purges come from a government controlled by one party in an area heavily populated by voters of another party.
So it is not cut and dried. There are reasonable arguments on both sides, but my personally calculation is it is less likely someone gets away with finding people who have moved and casting votes on their behalf. It is more likely that a bunch of politicians purge voting rolls in districts that lean the other direction. Especially as we have 150 years of evidence of the latter but few examples of the former.
I think that may have been after just a few years of not being able to purge voters due to lawsuits from the ACLU, so there is definitely a big enough pool of registered voters who are likely no longer living in that state to potentially swing elections at least some of the time. If I remember rightly, some recent key swing states have a lot of outward migration compared to the victory margins.