Rational miners realize that forcing a protocol change that isn’t acceptable to users or merchants will destroy valuation of the fork they choose to mine, so there is discebtive from such behavior.

But yes, if they do it anyway, there will be period with very slow blocks and that will hurt the project a lot but then difficulty will adjust downward.