From a user space process programmer's perspective the graphics device is some abstract thing, represented by the operating system through a unified API.

When it comes to actually setting the framebuffer mode on the hardware, well, in theory it sounds nice to have a common hardware standard like VESA to support this. But then such a low level interface was of little use to user space applications running in memory protected environments.

For a long time the X server was required to be SUID root because it drilled a hole through memory protection using ioperm so that it could talk to the graphics chip directly; but talking VESA required some code of the Video BIOS to execute, which technically requires a real mode environment the X server also included a 8086 emulator to run the Video BIOS code in. We had to live with this mess until KMS came along.

From a programmer's perspective KMS is the far nicer, much less complex solution, even on the low level. Yes, it requires dedicated code for each kind of GPU, yes there is some code duplication. But the advantage is a huge reduction in complexity: Not interacting with a Video BIOS (or a EFI driver) means, that you don't have to provide a runtime or execution interface in your kernel for them to operate in. Writing a universal emulator/VM, verifying that it always does the correct thing is much harder, that punching down a few dozen lines per GPU class to deal with the low level mode setting stuff.