Physical analogies don't really work in this situation because of the scale, and the payout.
A physical master key for a building has a few hundred thousand/a few million people that could potentially access it. The payout is low (i.e. the motivation is low on average)
An encryption backdoor to phones has a few billion people that could potentially access it. From anywhere in the world. The payout is huge (access to all iPhones).
Multiple entire governments would dedicate tens of millions of dollars and thousands of people to gain access to a ubiquitous backdoor on something like a phone. The same just isn't true with your building analogy -- they are completely different.
And put the HSMs where? In a big room? Protected by a door, with one key?
Or maybe each of the 650 elected politicians gets to hold 10 HSMs, each holding 10,000 keys? That way, by distributing trust, we can be completely sure that politicians are always snooping on about 10% of everyone's mails, instead of worrying about whether it's 100% or 0% at any particular moment.
Security *around* the master key is entirely about pinning liability to one human being at a time. Security through hot potato.
Have you considered that the locks need to have a weaker security if a key must exist which can open all the doors in the building?
The last part is probe to discussion, though. You need to make the lock weaker, yes, but maybe just a little bit and you maybe need 4.5 days to open it vs. the 5 days when there is no master key. It is a matter of math (mechanics in that case) and risk assessment.
If you consolidate security into a singular "skeleton key" - you 100% weaken your security.
It requires roughly half the picking effort.
in a lock you have multiple sets of pins. the key pushes pins, and if it pushes all the pins so that the top of the pin is at the boundary of the lock (the shear line), the lock turns.
There is a spring that pushes a connected pin down, which is what actually prevents the lock from turning. These are called driver pins. there is a separate pin(s) that the key actually interfaces with. The key pushes the pins until the driver pin moves past the shear line, when all driver pins and key pins are not interfering with the shear line, you can rotate/whatever the key and it is unlocked.
A master-keyed lock has additional discs inside the keyway, usually below the normal pins (closest to the key.) The discs are added based on the amount of extra movement needed to accept both the non-master, and the master key. So a master keyed lock has two, separate shearing points, the top of the regular pin, and the top of the master disc. This means there are at least two set-points for picking to get the driver pin out of the way - where the driver pin is flush with the shear line (as it would be with a regular, non-master key,) and where the normal pin's lower face is flush with the shear line (as it would be when a master key is inserted).
That’s not really true is it? If I have a building where every room has its own key, but there is also a “master key” that can open all doors; then it’s not “insecure”. You want to be pretty bl—dy careful with that master key, sure, but the idea isn’t crazy.