Interestingly, chatting with a physical passenger in the car is an entirely different level of distraction, from trying to chat with someone you can't see on the other end of the phone. For some reason (invoking visual synthesis to imagine how they are responding?) the phone case takes quite a bit more mental effort.

Try putting 'blinders' on when talking to a passenger and driving normally. E.g. put a hand up so you can't see them, even out of the corner of your eye. This A/B test (for me anyway) ups the mental effort 2X or more.