This is a very fair concern, and I sympathize with it a lot. I have thought about this a fair bit, and thought about eventually turning this into a "source-available" model (but not yet, I'd at least like to be fully launched on both App Stores first and have some user-base). There are pros and cons to that. At the end of the day, if you install an app from the App Store, you place trust in the developer. App Store Review only goes so far (e.g. look at Epic sneaking in an alternate payment system). Even if I made source available, the typical user has no guarantee that it matches the binary they're installing from the App Store.

I've also thought about starting with an OAuth scope that allows all operations except permanently deleting email, and asking the user to re-authenticate and upgrade their scope the first time they try to permanently delete something. This is a little awkward from the user experience, though. Maybe burying this as a fallback mechanism during the onboarding flow is an option.

General reputation-building is probably the right place to start.