We can absolutely fake that. I was at the WeAreDevelopers conference the other month, and one of the talks was a guy who had hooked up a voice-to-text engine, an LLM, and a synthesiser trained on his own voice, to WhatsApp — it was good enough that some people on the receiving end didn't believe it was a fake when he said so, and he had to prove that it was fake.

Combine this with the talking-heads AI used in things like:

https://youtu.be/K_XwseDwmuQ?si=DnszZFKVpSMMT0an

And: https://youtu.be/O0LGkwDw0Y8?si=mD_Qaf8pLODpg7FR