I think this is what many people think that Docker is, and what most people actually want, v. the application-centric approach offered by Docker.
However, GPUs are increasingly supporting virtualisation and multiprocessing. For example, nVidia supports multiplexing a single GPU across multiple VMs for VMWare View. I'm not sure if there is special driver paravirtualisation magic happening but I suspect so. But there is at least some GPU process control and resource scheduling now, and I expect this technique will be accessible to linux in the near future.
At present you can map through the host devices created by kernel drivers into the container, and then access them via userland libraries. You are still dependent on the library versions and the driver version matching. This works for CUDA and OpenGL. https://blog.simos.info/how-to-run-graphics-accelerated-gui-...
Also, the nVidia GRID licensing makes me vomit in my mouth.
It also requires the drivers to be installed and loaded by the host.
Lots of problems with the multi user use case, such as many video cards can be used to DMA main memory and also potentially affecting the card permanantly in some way ahead of being used by another user later.
For most purposes and running most applications, you can create an LXD and login to it and it feels just like a VM but with no VM overhead and you can see all the processes inside container from outside the container ("on the host"), e.g. in top.