Long theory short: it's the same reason the Beatles mono album versions sound a ton better than the stereo versions, because that was the optimized version because that's what people listened too at the time. Sound engineers aren't being paid to optimize the stereo or even 5.1 versions of the audio, only the theater version in whatever multi-channel standard is popular at the moment, so they don't bother they let the standard downmix stand.
My uninformed and probably stupid thoughts:
Hollywood was doing some amazing movies in the 90ish (think Seven, The Matrix, Requiem for a Dream etc). It was a mix of new and established players that were allowed to innovate and be creative. Then Hollywood became more risk averse, I assume because sales were unpredictable and production companies wanted guarantees similar to other industries' large investments. So they started recycling ideas, produce sequels and trading IP rights for dudes in specifically colored spandex suits, which came with a guaranteed fanbase that would watch anything with the brand. This eventually became the MCU-type of crap where everything is a 4th wall joke packed with references to current events, leading to a shelf life less than a year and plots became an afterthought. The sales mindset went as far as putting stuff in the movie for the sake of producing a compelling trailer. At the same time TV was taking the Hollywood's lunch money, but for some reason they doubled down on the approach and blamed streaming for their failures.
The issue isn't the format though, or that everyone has Netflix on their microwave display now. There are still great movies today, and if you get a chance to watch one in a good theatre it's a mesmerizing experience. Me and my partner watched The Joker, 1917 and Dune in Alamo Drafthouse (a theater that has zero-tolerance to texting/talking and serves food & drinks). It's an experience that trumps even great TV shows, when done right. It's just sad that it's so rare.
Covid caused a lot of smaller cinemas and chains high in debt to shut. But a lot of them are still going.
Cinemas are pivoting more to the luxury experience rather than just being the place you watch new movies. So they will get bigger screens, better projectors, better audio, etc. It's already started near me.
And before someone says "go build yourself a home cinema": this is the UK. There's no such thing as "free space" in most homes.
Just the other day I was helping a relative who had gotten a soundbar because her TV's speakers broke. She was complaining the sound carried too far to other rooms. I found the equalizer settings and turned down "woofer" which seemed to help, but then I read the manual that it only applied to an external subwoofer so I was left wondering if it was the placebo effect. Turns out there was a wireless subwoofer placed some distance away. I showed her the equalizer settings if she wanted to turn it down even more, and told her she could also just unplug the subwoofer if she wanted. To her, the inclusion of that box was actually an anti-feature.
And for other frequencies, I'd say middle-cost "prosumer" gear generally exacerbates dynamic range problems, being decent at low levels but distorted at higher levels. My Thinkpad speakers sound like crap all the time, and my receiver+speaker setup handles higher volumes without sounding louder due to distortion.
I still find the dialog way too quiet in most movies and 5.1 TV shows, even turning the center channel up +6dB.
I could turn it up even more but it's ridiculous that it should need more than that.
Some audio workstations (Logic) have "binaural" processing to try and emulate this effect. It can never be perfect because everyone's ears are shaped a little different.
About the pinna, a fun way to demonstrate this is to have someone close their eyes, then you snap or jangle keys at various locations and have them point. Then tape their ears to their head and repeat. They will be way off in the vertical. We can do left right-ish (with coning error) without the outer ear. But up down is impossible.
3D phase changes occur and create comb filters at you ear and your ear is unique to you.
This is why Kunstkopf(dummy head) recordings were never as successful as anticipated.
The explosions can be heard by my neighbors yet I can barely hear the dialogue at the same volume.
Find myself basically doing half the mix work myself with the volume control. Too many audiophiles in the production chain focusing on something that barely matters at the expense of most viewers.