In a typical linux distro all apps are run under the same user which means they can do whatever they want to each other and user's files. So X server being secure or not doesn't really change anything.

By the way Android, unlike Linux, runs every app under a different user account.