This implies that the right can be denied for other reasons, which would mean that it's not an inherent right of citizens. And, indeed, courts do not consider e.g. restrictions on voting for citizens with criminal records to be unconstitutional.

And it's not like it's a new invention. I mean, women couldn't vote back when 15A was passed, and they still couldn't vote after. It wasn't a surprise, either - many suffragists opposed 15A even before it was ratified for this exact reason, because its wording made it clear that it wasn't covering sex.