Xauthority isn't fine-grained. Once you get a cookie, you haven't any restriction to what you can do with the X server.
Re:Malware Found in the Ubuntu Snap Store
by mikedelfino on Sunday May 13, 2018 @10:20AM
(#17059012)
We could generate an "untrusted" cookie. This prevents clients using it from meddling with "trusted" clients.
It's not really fine grained and also doesn't prevent untrusted clients from meddling with one another, but seems like a starting point for someone inclined to add more security to X.
Of course you should also prevent the program from reading the original privileged Xauthority data. Running it as a different user does the trick.