Things aren't so simple. A tax (tariff in this case) negatively affects both sides of a transaction. The effect on our side is then mitigated because the money from that tax lets us reduce other taxes in our own country.

So yes, in the end it does work out to be a penalty to the origin country.

The only local harm relates to marginal utility, but this is offset by the benefits of national security and improvements to the mix of different types of employment.