The catch is that the difficulty adjustment only happens every ~2k blocks. If a mass exodus of mining capacity happens really quickly, it could leave the network stranded. But if they leave at a reasonable pace, it will ramp down. That’s why I said “hopefully” before.