Your votes are being aggregated, that's what the karma counter is all about.

But 'high karma' as such shouldn't be a factor, it can mean many things. Average karma per comment would be a much better way of judging peoples contributions.