Because you're going directly off of the vote, instead of a representation of that vote via the elector for that area. It's a subtle difference, but one that brings to light some of the fears with direct election.