Actually, backtraces work correctly without explicit framepointers (in a typical GNU environment using ELF+DWARF).
The general concept has existed in DWARF since version 2 in 1992. The mechanism used for this is known as Call Frame Information (CFI)[0][1] — not to be confused with Control Flow Integrity, which is unrelated.
Here's some example libgcc code that evaluates CFI metadata[2]; there's similar logic in the libunwind component of llvm[3].
Burning a register on a frame pointer is a big deal on i386 and somewhat less so on amd64; there are other platforms where the impact is even lower. So, just know that you don't have to include FPs to be able to get stack traces.
If you're interested in how to apply these directives to hand-written assembler routines, there are some nice examples in [0].
[0]: https://www.imperialviolet.org/2017/01/18/cfi.html
[1]: https://sourceware.org/binutils/docs/as/CFI-directives.html
[2]: https://github.com/gcc-mirror/gcc/blob/master/libgcc/unwind-...
[3]: https://github.com/llvm/llvm-project/blob/main/libunwind/src...
It's hard to optimize only if you accept the tenet that it sys._getframe(), and all its uses, must continue to work exactly the same in optimized code.
Instead, you can just declare that that it (and any related sort of anti-pattern of the same ilk) won't work in optimized code. If you want the speed from optimized compiling of some code, then do not do to those things in that particular code.
The programmer can also be given fine-grained tools over optimization, so as to be able to choose how much is done where, on at least a function-by-function basis, if not statement or expression.
It's not written in stone that compiled code must behave exactly as interpreted code in every last regard, or that optimized code must behave as unoptimized code, in every regard. They behave the same in those ways which are selected as requirements and documented, and that's it.
In C in a GNU environment, I suspect your Glibc backtrace() function won't work very well if the code is compiled with -fomit-frame-pointer.
In the abstract semantics of C++, there are situations where the existence of temporary objects is implied. These objects are of a programmer-defined class type and can have constructors and destructors with side-effects. Yet, C++ allows compete freedom in optimizing away temporary objects.
The compiler could help by diagnosing situations, as much as possible, when it's not able to preserve this kind of semantics. Like if a sys._getframe call is being compiled with optimizations that rule it out, a warning could be issued that it won't work, and the generated code for it could blow up at run-time, if stepped on.
One way in which compiled code in a dynamic language could differ from interpreted code (or less "vigorously" compiled code) is safety. For that, you want some fine-grained, explicit switch, which expresses "in this block of code it's okay to make certain unsafe assumptions about values and types". Then, he optimizer removes checks from the generated code, or chooses unsafe primitives from the VM instruction set.
The code will then behave differently under conditions where the assumptions are violated. The unoptimized code will gracefully detect the problems, whereas the vigorously compiled code will behave erratically.
This entire view can nicely take into account programmer skill levels. Advanced optimization simply isn't foisted onto programmers of all skill levels, so then they have to grapple with issues they don't understand, with impaired ability to debug. You make it opt-in. People debug their programs to maturity without it and then gradually introduce it in places that are identified as bottlenecks.