If you don't have enough to pay your taxes, then you're in a very unusual position, because the average American gets their taxes withheld from their paycheck regularly. It must mean that you own your own business, or are doing something fairly unusual, and have failed to properly budget with taxes in mind. No one "doesn't have enough to pay taxes" just because they don't make enough money, because income tax is progressive specifically to avoid that type of problem.

As for it being "your money"....sure, one can say it's "your money", but you owe it to the government for services they provide on an ongoing and pervasive basis. It's like a subscription fee for civilized living.