The code generation and execution is trivially problematic. I don't need to try it out to know that if someone can coerce one of these bots to spit out its prompt, someone could coerce it to execute dangerous code if it's wired up to an interpreter.
As for the other part of my analysis, in order to empirically test it I would have to build a full-featured voice AI, release it to millions of users, and see if I was able to predict everything they tried to do. Given that my prediction is that even Apple can't do that well, I'm not sure why you think I would bother to try it out.
If an engineer told you that a bridge would collapse if built a certain way, you wouldn't insist on trying to build it anyway just to be sure. Most of the time in engineering you can't run a full-scale test, you have to make do with analyses. If you want to critique my analysis, I'm all ears, but "you haven't actually tested it" is not a critique.
All of my analysis is based on my usage of ChatGPT and my understanding of the underlying model. The hype is largely driven by people who don't understand how it actually works and think they're interacting with an artificial general intelligence.
It's a very impressive language model with a lot of applications, but with many fewer applications than the hype would suggest.
I'm hyped because I threw normal text (sentences) against it and it always did what I wanted and it didn't matter if it was bad English or bad German.
I didn't even try promt hacking or promt tuning because natural language already worked so good.
It's a tremendous good 'normal human' text interface.
A much safer approach would be to have it produce a JSON object that is interpreted by the wrapping code. This allows you to inspect the interpreting code and provide guarantees about what can and cannot be done by users.
But neither execution model solves my main point: someone still has to write the interpreter (or the library functions if you go the dangerous route). Someone has to think ahead and guess what the user is going to try to do and provide APIs that allow that. And that work is most of the work: our existing intent recognition models are actually not bad at all at their job, the failings these voice assistants have are almost all to do with predicting the ways in which users will try to interact with the system.