It's desirable, but typically hardware devices have DMA-capable buses that can read / write to arbitrary physical memory. So a buggy or malicious WiFi driver may be able to use the WiFi DMA hardware to write into the GPU driver's memory, and no pure software mechanism can stop it from doing so.

IOMMUs solve this problem by giving hardware devices their own virtual address spaces.