some peripherals like the dram controller are rigged up so the arm can never control them

others like the 2d core default to the same mode, but can be re-configured to allow the arm to control it

there may be more peripherals that behave the same way, and require code on the VPU to help out

and like others said, changing the arm freq, from the arm core, could be dangerous, it may just lock up, because it turned its own clock off