You need to be running a GPU driver on the host that supports the container cuda version.
So in theory yes, in practice, weird issue occur sometimes that really suck to debug. For example why do I get NaN loss after spending 8 days on 128 GPUs with this specific set of drivers+cuda container? (Don't hold it that way, use a matching cuda version...)
Also a lot of data scientists HATE sys-admin tasks and docker falls squarely into that for many people.
You need to be running a GPU driver on the host that supports the container cuda version.
So in theory yes, in practice, weird issue occur sometimes that really suck to debug. For example why do I get NaN loss after spending 8 days on 128 GPUs with this specific set of drivers+cuda container? (Don't hold it that way, use a matching cuda version...)
Also a lot of data scientists HATE sys-admin tasks and docker falls squarely into that for many people.