As someone else mentioned, going to and from fiat has always been an issue for crypto. So as a result, exchanges exist. Problem is with exchanges in order for settlement to happen the money has to be held somewhere where it can actually be used to settle trades. That is, users can't just keep the tokens in their own wallet until the trade is complete then pay up (or they could just not pay if eg; the trade was unfavorable... for example if they could get a better price now).

Since crypto transactions routinely take 15+ minutes you can't just pay right as you trade. And if using fiat to buy crypto then there's no getting around the exchange needing to hold your fiat.

All that said, exchanges need proper auditing and oversight. Building exchanges is a slow process. This is something that coinbase actually got right but alas they were outglossed by binance and FTX that offered very risky offerings like leverage, way more coins, etc.