I've echoed this sentiment from time to time and someone recently asked me "Why don't you just disconnect and apologize later?"

Might not work for you, but I found this surprisingly liberating: allowing myself to just drop the ball, not respond, or just say "Sorry I can't" with no justification, so I can focus my energy where I need it