What I do here (https://github.com/antirez/zx2040/) is: stay overclocked when not accessing the flash, downclock when accessing the flash via memory map, return back to the higher clock. Works without issues. I find this method to work more reliably than dividing the clock in practice. Maybe I did something wrong dividing the clock so it was not stable, but the system I described really works flawlessy.
If you're playing with overclocking the RP2040 it's good to know that the flash chip might not be able to keep up with those higher clocks though, which leads to inscrutable crashes, and of course the official documentation doesn't endorse overclocking so they don't tell you how to work around that. The flash runs on a configurable clock divider defaulting to 4, so if you're running the CPU at double the nominal clock you can raise the divider to 8 to keep the flash running at its original speed.