> Do you think the ai is really running that command?

Yes, they're calling it "agentic" AI or "tool use".

> or could run that shell command with appropriate privileges?

Yes, it's called humans doing stupid shit when using dangerous shit. Happens daily.