This is kinda possible already today with the Kitty graphics protocol, I made a demo here of rendering 3D graphics[1] with kitty. The actual important missing thing (and which ratty seems to also not include) is vsync.

If rendering is not aligned then it's possible for the terminal emulator to read the framebuffer while the application is writing to it, causing visual artifacts.

[1] https://x.com/zack_overflow/status/2035921425341763756?s=20