I think what would be helpful is to have an option to ignore all voting for purpose of sorting, so that you can vote if you want to, but if you don't like to, you can disable that option and it will display only in the chronological order (it could hide the voting options too if you wanted to, but I suppose it doesn't have to do).