If you have the kind of clueless/toxic manager who is going to abuse this sort of arrangement, they are probably never going to agree to this sort of arrangement in this first place.

For nearly all working relationships, I think it's reasonable to keep an emergency communication channel open.