Maybe because just speaking on the phone in traffic makes a lot less attentive so you're prone to create accidents.
There are worst things like eating or talking to someone else in the car. (look at the road not the person when you're talking)
https://psycnet.apa.org/record/2008-18024-008
>The results indicate that passenger conversations differ from cell phone conversations because the surrounding traffic not only becomes a topic of the conversation, helping driver and passenger to share situation awareness, but the driving condition also has a direct influence on the complexity of the conversation, thereby mitigating the potential negative effects of a conversation on driving.
Yes, when talking to someone in the car they should be aware of the traffic and pause the conversation, where you have to tell the person on the other end of the phone that you need to go.
That affects your attention on driving.
You do appreciated the relative silence (no one on their phone or talking above a whisper) on trains in Japan.
Brisbane trains on the other hand - you need to wear noise-cancelling headphones (music optional) to block out other people's conversations - even in the quiet carriages.