> I don't know what is holding back container/VM access to graphics hardware, but it can't be insurmountable if the cloud providers are doing it.

Cloud providers have graphics hardware with SR-IOV support. It is exactly the kind of functionality, that the GPU vendors use for segmentation of their more expensive gear.