Even if you don't, your required precision blows up very quickly, and your performance drops.
In more general context, you could choose e as the base of your number system and then multiplying by e is a trivial shift. Then, however, multiplying by 2 would make arithmetically no sense. The natural question that arises is whether it's possible to represent all computable numbers non-lossy in one system. The answer is yes.
Besides that, I think you should re-evaluate your decision-making for votes.
Yes. that's kind of my point: Either you have a finite precision cutoff and lose accuracy, or the memory used for your arbitrary precision representation blows up to incredibly large values (transcendentals will do this to you quickly), or a combination of the two.
However, it's possible using MPFR in a way without suffering this issues. One could increase precision every time there would be rounding.