Each of those people would have to do some sort of authentication. Each of those mechanisms is just as subject to accidental disclosure, hacking, etc as any other.

Make a door, it won't stay secret. And you can't assure that there won't be bad actors involved in any number of parties that have to agree.

The whole backdoor system, once it exists, is open to subversion and/or misuse. The only way to not have the problem is not create the door in the first place.