I used the term "algorithm" as a crutch, but it seems likely there are tools out there than can do fantastic keyframe & data rate shaping.

The most apparent is whatever Apple's been using for years for their movie trailers. Not a single artifact, low data rate, etc. It's better than your average 2-pass. But who knows, the "magic" could simply be to start with uncompressed source...