Ok, here's a scenario I have seen implemented before when dealing with HSMs and root CAs for a fairly serious PKI. You set up a 3 out of 5 share with each of the 5 keys stored on separate hardware tokens. The tokens are physically secured in safes, in tamper evident bags. To perform operations on the HSM, three out of the 5 tokens are required. The key custodians belong to different teams who can only open the safe for their token.

Now, you need to compromise at least 3 safes to do anything on the HSM. If a token is lost, stolen or damaged, there are still 2 others which can be used. Loss or theft of any two tokens does not compromise the HSM. Any unauthorised access to a token can be detected due to the tamper evident bag.

This gives you strong protection and assurance against any malicious insider or attacker. Having a single key (even if protected in a safe) is much weaker, and carries the risk of key loss (so you would have to have token backups, multiplying rather than dividing the risk).