Generally, no. Often you’ll see GPUs have extensive access to the system’s physical memory and vice versa which makes exploiting either processor from the other fairly common due to buggy drivers.