That is missing the point. Arrests are not that common either, but the mere possibility of them does change behaviour. The chargeback is an important part of credit card transactions - it makes the card-holder more likely to spend, and it makes it more likely that the payment recipient is going to deliver.

I have both issued chargebacks for non delivered goods, and had chargebacks raised against me as a merchant. Generally fraudulent chargebacks are easy to defeat if you run your business well. Generally chargebacks are easy to win if you are a sensible consumer. The costs of chargebacks on consumers and vendors alike are vastly outweighed by their benefits.

Going to a commerce system where that layer of trust is removed is going to be more difficult to do.