This is why Apple (and Google) is in a privileged position to tackle this issue at the OS level. If you currently trust your OS, then having a local agent use your apps wouldn't be terribly different (prompt injection risk aside)