You don't need X, and I wouldn't assume its remnants of userspace mode setting would work on non-x86.
The amdgpu kernel driver POSTs the GPUs when loaded just fine! I run an RX 480 on a MACCHIATObin with FreeBSD and Wayland only :)
Plus, there are two ways to get the GPU in UEFI already – X86EmulatorPkg and native aarch64 GOP driver builds provided by AMD.
Pretty fascinating that X11 of all things is dealing with Video BIOS initialization, though. The sheer amount of functionality that was overloaded into X...