Locations of prisons tend to be rural and have very different demographics from the places where prisoners would otherwise be living.
Deciding to let prisoners vote needs to be followed up by a decision on whether to let them vote as if they returned to their former home, or in the location of the prison.
Their vote should be able to impact the conditions of the prison
But another way to look at it, is that by being counted as part of the population in the vicinity of the prison, it provides an incentive to have prisons and to incarcerate people, and it also reduces the representation of the places prisoners come from.
The prisoners make up about 0.5% of the population of the US. That's bigger than the margins we saw in several swing states in the last few elections.
I don't even care if somebody is currently in prison; I think every prison needs a polling place and facilities for on-site journalists/reporters who are covering what is going on inside.
If your elections could be significantly moved by the votes of prisoners, you live in a failed state.