They're mine.
If there are naked shorts floating around (or shorts covered by my stock without permission), someone else is making money off of risk I am taking on, but didn't agree to. There can be a cascading set of failures which lands with me not having my shares.
It's the difference between taking out a $300,000 mortgage on a $500,000 house, versus borrowing $300,000 with no collateral. You'll get a different interest rate, if you can get a loan at all. And this resembles someone taking a $300,000 mortgage, only missing the house.
Yes, there's always a risk, but that risk profile is very, very different (esp. in the case of catastrophic events, like a stock market collapse or similar, when many institutions might be going down at the same time).
If you don't want that, don't. It's in your control.
Even during the heights of the last few financial crises, clearinghouses did not fail.
(Option writers and people trading futures are in a very similar situation to short sellers. They also have clearing houses.)