It is possible to do this but it's considered a bit non secure. So for your desktop it's fine, whether it's a good idea in a multi user environment is another story.

It also requires the drivers to be installed and loaded by the host.

Lots of problems with the multi user use case, such as many video cards can be used to DMA main memory and also potentially affecting the card permanantly in some way ahead of being used by another user later.