So this is not just theoretical, at the same time: you have no idea what closed model vendors are doing either, and there they could swap out the model after delivering a payload so you'd lose the opportunity to audit it to see what prompts will trigger what kind of response.
https://www.crowdstrike.com/en-us/blog/crowdstrike-researche...
So this is not just theoretical, at the same time: you have no idea what closed model vendors are doing either, and there they could swap out the model after delivering a payload so you'd lose the opportunity to audit it to see what prompts will trigger what kind of response.