I haven't tried XQuartz at all because I didn't think that kind of software has plenty of resource and is well maintained. Retina support is just one problem caused by lack of maintenance.

There are only two options: hack Virgil 3D or hack XQuartz. You cannot "just" run XQuartz and solve problems. And I choose hacking Virgil 3D because it should have less communication overhead and work with Wayland.