You can generally allocated GPU buffers without the window manager. So to be effective this would have to be in the kernel driver. Opengl/GPGPU is generally done in the user space driver so it'd still be separate.