Yup, I'd strongly consider something similar. But I'd have to seriously consider the security implications: giving Windows access to the graphics card could end up being more hassle than it's worth.

GPUs' memory management isn't exactly security friendly.

-> https://arxiv.org/pdf/1305.7383.pdf