Then draw it with the CPU? Alternatively, have this cache live in the GPU as a texture, where dedicated hardware can handle most of the transformations.