> That way no matter how badly my program misbehaves, it will not be able to access memory outside of the sandbox
The problem is not about memory outside of the sandbox, but inside. Please read about return-oriented programming, for example, where a buffer overflow bug of a process can be exploited to hijack said process to do work it was not meant to do normally. If this error happened for example in sudo, it can very well be used to gain privileges and do real harm — and this whole problem domain is almost entirely solved by using a “safe” language.
Even with a safe language, there's vulnerabilities like supply chain attacks which allow malicious code to use an escape hatch to access memory outside of the process. I.e. I could be programming in Rust, but one of the crates I depend on could silently add an unsafe block which does nefarious things. OS/hardware level sandboxing could prevent many such classes of exploits.