Isn't difficulty set based on the previous block time?

No, the difficulty is updated every 2016 blocks. This should be once every two weeks, but if the worldwide hash power halves it'll of course take four weeks until it can update. The difficulty adjustment is capped to +300% / -75%.

The issue here is that the Bitcoin reward has to pay for the mining cost, but there is nothing preventing the Bitcoin price from dropping lower. So miners will either have to mine at a loss for (on average) a week until a difficulty adjustment, or turn off their equipment. Turning it off will make the wait even longer and the mining price per block will increase even more for the remaining miners - leading to more of them operating at a loss or shutting down which makes the problem even worse.

If the difficulty is too low then the network becomes vulnerable to a 51% attack. If it's too high then it becomes less profitable to mine and transaction processing speed slows down. So it has to be high enough to prevent attacks, but low enough for mining to be profitable.