What happens when your trusted code gets exploited? That's also the reason why we have overhead in the form of ACLs, memory protection, ASLR, packet filtering, deprivileged processes, seccomp sandboxes, etc. even though in principle we're all running "trusted code".

Not saying that there isn't a scenario where it would be ok to turn that off, but it's the exception.