Because it is weird to force people to do things that they don't want to if it doesn't harm another person. In this instance, forcing a vote can actually harm others if someone votes randomly.

The idea behind freedom is not absolute freedom. It is a balance of the contract with society vs individual freedom. You want to maximize individual freedom, but for everyone (not a singular person). Clearly things like the "freedom" to murder someone doesn't outweigh that person's individual freedom to live and thus we make laws (and ways to enforce them). Mandating voting decreases individual freedom and has a potential for reduced societal freedom (while little to no real potential for increase. Much prone to populist candidates). So why mandate? Don't solve problems that aren't problems. If someone doesn't want to vote, how does that harm you?