I did that. Customized Pi to run in Bubblewrap, made an action authorization flow, etc. But that's throwing the baby with the bathwater. This gives you a harmless, less capable agent. To get both safety and usefulness, Ant's solution of a fast classifier works really well.
For me, its the other way around.
Strong sandbox + disabling permission checks gives me maximum capabilities, least annoyance and (relative) peace of mind that any prompt injection, malicious plugin,... won't wreck my shit or leak credentials of any kind
Wrap your agent of choice in e.g. docker sandbox, nono.sh, Nvidia OpenShell, microsanbox or any other sandboxing tool and run it without any checks (i.e. "yolo mode", dangerously-skip-permissions etc.)
IMHO it makes the whole category of problems around permission fatigue, trusting the harness (or plugin) authors etc go away.