The personal violation you feel as a victim of a mugging or a burglary is more visceral and real, whether someone steals a dollar or ten thousand dollars. The perp takes something from your psyche, and we want to extract the same.

When some character fudges a spreadsheet or sets up a viscous cycle in business, it is more abstract. We’ve all done it to some degree (received an undeserved discount, more change than owed, etc), and the black and white “wrongness” varies.

Another consideration is that the punishment for white collar crime is very severe in terms of loss of agency and autonomy. Once you’re convicted of a felony, you’re kryptonite — you’re trading your comfy middle class existence for something riskier and worse. Typical suburban punter is losing most of your friends and probably your home and spouse. The kids lose their nice school or get stuck in a grubby apartment. White collar criminals have more to lose, so the punishment standard is different.

There's a nice line by Terry Prachett in his book "Going Postal". This is near the start of the story, as the protagonist, a conman, is being confronted with the effects of his cons.

> "I Worked It Out. You Have Killed Two Point Three Three Eight People," said the golem calmly.

> "I have never laid a finger on anyone in my life, Mr Pump. I may be–– all the things you know I am, but I am not a killer! I have never so much as drawn a sword!"

> "No, You Have Not. But You Have Stolen, Embezzled, Defrauded And Swindled Without Discrimination, Mr Lipvig. You Have Ruined Businesses And Destroyed Jobs. When Banks Fail, It Is Seldom Bankers Who Starve. Your Actions Have Taken Money From Those Who Had Little Enough To Begin With. In A Myriad Small Ways You Have Hastened The Deaths Of Many. You Do Not Know Them. You Did Not See Them Bleed. But You Snatched Bread From Their Mouths And Tore Clothes From Their Backs. For Sport, Mr Lipvig. For Sport. For The Joy Of The Game.”

White-collar crime has effects. Every time people are encouraged to clock out, then clean up their station, that's 15 minutes of wage theft. Heck, sometimes this becomes so common that the criminals work to make it legal. Every time that Amazon has their mandatory daily exit screening, that's 30-60 minutes of time that should be on the clock, but is completely unpaid, legal and SCOTUS-approved, and wildly unethical.

Let me unpack this. Do you think any given person is more likely to be considered “kryptonite” because they committed insider trading than because they did, say, a smash-and-grab? If not, your last paragraph is not justifying more lenient punishments for a certain class of crime, but for a certain class of socio-economic status. To put it in other words, only poor people should go to jail.

The same reasons lie behind why we also tend to ignore many external costs.