These scams are imaginary. Yes they have probably happened but it's far more likely that your mother will get scammed by a "legitimate" financial advisor than some stranger using an AI to impersonate you.
And no need for AI. I already had family members getting scammed by facebook or mail impersonation.
But at some point, the capability for AI is going to get so cheap it will be mass-produced by bots from millions of identities like SMS or email scams.
It's not an "if". It's a "when".
I'd rather have my mother at least have the reflex to try to send me a message if I can't remember the passphrase to check if it's indeed me.
And if it doesn't, work, it doesn't. I'm not working for a 3 letters agency and my mum is an old tech illiterate lady. There is a limit.
A passphrase is cheap. If you never need to use it, so what?