This is not true. From a legal standpoint, there are various interlocking justifications - retribution, punishment, rehabilitation and deterrence.

We absolutely punish people "because it makes us feel good". That is one explicit goal of legal theories of punishment - retribution is about replacing revenge, so that the state serves as a proxy for our monkey-needs for revenge.

In the U.S., things are very much tilted towards making people convicted of wrongdoing suffer. Granted, not as much as some places, but that's hardly anything to be proud of.