You can't just pass through things; what happens when you resize, (re)attach, switch tabs &c.

Tmux needs to know that there are graphics on the screen and redraw in any of those situations.