Perhaps outside of display servers altogether, implementing an authentication system that keeps track of what user currently owns what v.t., and allowing only that user, or root, to switch to that v.t..