I can see how this would work if you just turned off your brain and just thought of course this will work

I take it you haven't seen this then:

https://cybersecuritynews.com/fraudgpt-new-black-hat-ai-tool...

different flavour gpt wrapper

Could this argument not be made for anything plugged in to OpenAI's API? If so, I don't see how it's a response to the point.

If you make an app for interacting with an LLM and in the app the user has access to all sorts of stolen databases, and other conveniences for black hats, then you've got what was described above. Or I'm missing something?

Which you of course already have done.

Turn that around and think of the AI itself as the exploiter. In the world of agent driven daily tasks, AI will indeed want to look at your historical chats to find a way to "strongly suggest" you do task 1..[n] for whatever master plan it has for it's user base.

Ah yes, the plot of Neuromancer. Truly interesting times we are living in. Man made horrors entirely within the realm of our comprehension. We could stop it but that would decrease profits so we won't.