I wonder whether a "bridge-across-the-void" approach would work. The idea is that a speculative optimization would not discard its original form, so that deoptimization is as cheap as a rollback. Then, when speculative optimizations are performed, they form a kind of bridging branching structure which reaches across the void of invalid low-level IRs, trying to find a safe point at which the optimization can complete.

I know that Truffle is based on partial evaluation; how close are you to Futamura #2? That'd relieve you of the burden of having to care about what an "interpreter stack frame" even looks like. OTOH I can confirm that the same kind of speculation in optimization has to occur during self-application.

It really is a thorny problem, isn't it?