There is still only one kernel, the host if you will. Since graphics drivers are kernel mode, it has to be loaded into the kernel. So VM style pass-through of PCI devices doesn't really make sense.

However, GPUs are increasingly supporting virtualisation and multiprocessing. For example, nVidia supports multiplexing a single GPU across multiple VMs for VMWare View. I'm not sure if there is special driver paravirtualisation magic happening but I suspect so. But there is at least some GPU process control and resource scheduling now, and I expect this technique will be accessible to linux in the near future.

At present you can map through the host devices created by kernel drivers into the container, and then access them via userland libraries. You are still dependent on the library versions and the driver version matching. This works for CUDA and OpenGL. https://blog.simos.info/how-to-run-graphics-accelerated-gui-...

Relevant talk from GTC 2017. Note that a special GRID driver is required, and it must match the driver on the host (VMWare Horizon). http://on-demand.gputechconf.com/gtc/2017/presentation/s7349...

Also, the nVidia GRID licensing makes me vomit in my mouth.

It is possible to do this but it's considered a bit non secure. So for your desktop it's fine, whether it's a good idea in a multi user environment is another story.

It also requires the drivers to be installed and loaded by the host.

Lots of problems with the multi user use case, such as many video cards can be used to DMA main memory and also potentially affecting the card permanantly in some way ahead of being used by another user later.