Actually, if anyone has direct control, it's the Secretary of the Treasury. And as we learned from the confirmation hearings, he doesn't know how to pay his taxes any better than the rest of us.
This was Fromm's particular insight about modern industrial civilisation; in past social and productive orders, it was easier to see your oppressor, often in singularity. But now, not only is his nature infinitely more obscured, but, in fact, he is a legitimately anonymous, organic whole. Nobody in particular to hold responsible.
It is like any other big company. The call centre people are never responsible for the mess the company made to you. However, if you do not want to devote significant resources, you are pretty much screwed up and yell at the call centre people.
The difference is that, in the most cases, you can just forget about the corporation and live your life. There is no opt-out from the IRS, though.