Fundamentally, something better should exist and websites get the flak of bad iframe security. Annoyingly, any site with an iframe on it can't determine what locations are being accessed within the iframe to be able to accurately let there be a "report ad" feature that grabs the sites involved. Sometimes ad network iframes are nested 3-4 deep (disgusting but true) and it is impossible to read the location (yet these iframes can call window.top.location with ease).
I don't know what needs to exist but trying to work against these bad actors is nearly impossible for web devs (yet the burden is placed on the site).
Thanks for taking the time to comment - I understand it's very complex but I feel like this important subject hasn't seen a lot of real world use (we've decided against using sandboxing because it doesn't work for ads).
So, if as a normal user (someone that doesn't understand malware versus OS-level dialog) sees a big warning in their trusted browser (or a popup) without having clicked on anything at all they might be compelled to download some malware. So yes, getting sandboxed iframes right would have been amazing but in my mind it was a failure.
Those changes represented a pretty dramatic improvement and all modern browsers are now in alignment. But the resulting behavior does make numerous concessions for compatibility with existing content. Take your complaint about any child frame being able to navigate top. That behavior was retained for both content compatibility and because it was a necessary security measure.
To explain the security concern, standardized frame communication predates the broad adoption of X-Frame-Options. So, the only reliable, cross-browser click-jacking defense at the time was frame-busting via top-navigation. And because we're talking about the Web, you have to consider a transition path of many years for existing content.
It may not seem like a great explanation, but the fact is that most of the confusing security behaviors on the Web really are the result of being boxed in by the need to keep existing content working. And, unfortunately, it takes many years to get improved mechanisms widely supported among browsers and to then migrate the majority of the Web onto them.