Your third argument for punishment is "punishment"? Did you mean something else there?

I think the US criminal justice system frequently does make an effort to compensate victims where doing so is straightforward and a necessary part of "unwinding" the crime. For example, someone convicted of wire fraud typically forfeits their ill-gotten gains and where possible, those funds typically go to the victims.

There is a broader concept of criminal restitution, though you're right, more frequently, making amends to victims happens on the civil side.