https://en.wikipedia.org/wiki/Compelled_speech
EDIT: IANAL, please do not cite this as any legal justification in a court if you fail to file any form with a government.
If your income is subject to taxation, you can't refuse to fill out the paperwork on the grounds that doing so lends legitimacy to an institution you don't support. The same should be the case for voting.
Mandatory voting would have the same requirements. Anybody who previously could vote now must.
In terms of state power: it's not all that great of an expansion. If you make over the taxable limit, you pay federal taxes. If you own any sort of land, there's a good chance you pay local taxes on it. These sorts of compulsions apply to (equally?) large swathes of the population without necessitating a great deal of state power.
The closest thing to compare compelling people to vote would be the draft.
Like I said above, a mandatory vote does not require individuals to make a political statement any more than taxation does, unless it is coupled with a limited selection of candidates. We can talk about the propriety of a mandatory vote, but the legality of one in the United States seems fairly cut-and-dry.
You are legally compelled by the government to fill out tax paperwork every year. Being (much more lightly) compelled to fill out a ballot is no different.