(I do understand that source code isn't what ships in the binary, but for the sake argument let's say they're 1:1 in size.)
Yelp, for example, is what you might call a "straightforward CRUD app" (to Yelp engineers, I know it's probably legit complicated and hard), and that is 292MB on the App Store.
It's probably to do with how the framework handles lifecycle management and combining static assets like text and image with business logic that lives in Controllers.
These are code. Swift is a safe language with more runtime checks than other "zero-abstraction" languages. It also support "value" semantics and can deploy monomorphization for generics (although no guarantee). All these means you can have functions with slightly different view models duplicated many times throughout the binary.
Not to mention the language itself need to generate a lot of retain / releases for refcounting purpose (the blog post also pointed this out).
All in all, Swift as a language is not particularly optimized for small binary sizes, and there are a lot of trade-offs made to improve the usability rather than binary size. That has been said, there can be more opportunities exploited (and right now not) to reduce the binary size from compiler side.