Conversely, if there are so many prisoners or the disenfranchised that, in a democracy, they can sway elections and drive policy, isn't that a sign there's something wrong with the system, e.g. disenfranchisement and prison is being used as a weapon against parts of the population? Denying them the vote means the system is that much harder to fix.

On the other hand, if only those who really deserve it lose their right to vote (i.e. a small fraction of the population), then it doesn't really matter if they can vote or not - their small numbers mean they won't make a difference. But the benefit is that disenfranchisement can no longer be used as a political weapon.