Passphrases suck enough whenever you have to log back in. Are people really gonna put up with that every time they want to use their phone?
On the other hand, if there were a convenient way to toggle between passphrases and 4-digit unlock, (especially if you had to use the passphrases to toggle back to 4-digit) then I would be all for it.
The long passphrase is used to mount the encrypted file system upon startup, and the PIN code is used merely as a "screen lock", as a casual deterrent from your friends and casual thieves from swiping through your photos before you can catch them.
The file system can be automatically unmounted after a set period of inactivity, or if the user wants to unmount it on demand. After the file system is unmounted, the data is secure again and the long passphrase will once again be needed before anything can be done with the phone.
I currently use a generated long password on my Android phone and have adapted to the extra work, but having the option to enter a password once a day and a pin or shorter password throughout that day would be a welcome convenience option, and it's not really significantly more onerous than just a pin.
Low-friction UI has been Apple's differentiator for years. If you have to take the time to type out a secure passphrase every time you want to interact with your phone, people will stop interacting with their phones (or use phones that suck less).
Why do you think they built Touch ID? Because typing even a 4-digit PIN is too annoying!
Longer answer: There's a key that encrypts the actual data, and that key is stored on disk, but encrypted with your passcode along with a hardware key. The hardware key cannot be read, only used to decrypt. Changing your code just changes the key stored to disk, but not the encryption key, so it's quick, but preserves security.
Longest and most accurate answer: https://www.apple.com/business/docs/iOS_Security_Guide.pdf from page 10.