It's not X server that's important. The Android Linux kernel has different modules enabled/disabled. For instance, standard IPC is replaced by Binders, and kernel module loading is typically disabled.

What you need is a super-set of android-Linux and desktop linux with all possible conflicts resolved.