Compare it to the other servers in the same `hw` directory:

    ├── hw
    │   ├── kdrive
    │   ├── vfb
    │   ├── xfree86              <- this one is Xorg
    │   ├── xnest
    │   ├── xquartz
    │   ├── xwayland
    │   └── xwin

Xorg/xfree86 is the one that renders to your display hardware (metal), compared to `vfb` which renders to the Linux Virtual FrameBuffer API, `xnest` which renders to another X server, `xquartz` which renders to Apple's "quartz" display system, `xwayland` which renders to a wayland server, and xwin which renders to Win32.

And, uh `kdrive`, which renders to... um... something or other.

Edit: Also, while Linux now has Atomic ModeSetting and the Direct Rendering Manager APIs, for a lot of Xorg/Xfree86's history it didn't, and the X server did a lot more direct manipulation of the display hardware than it does now. Switching virtual consoles between X consoles and Linux terminal consoles used to be a weird dance between Xorg in userspace and the kernel to make sure that the hardware handoff between them didn't mess up. This dance took about a second to complete (which is forever in CPU time) rather than being the instantaneous sort of switch it is now. But even now, with DRM and the other stuff, Xorg still does a lot of talking directly to the hardware.