The number of miners does not really matter. Whatever miner mined a given block must receive enough to cover the electricity cost, regardless of how many other miners are involved. There is also no particular incentive for a miner to reduce his own electricity consumption in response to the difficulty being reduced. The only reason miners will shut their equipment down is if they are unable to cover their electricity costs because e.g. nobody is willing to pay the higher transaction fees.

Obviously everything I described is a simplified model where energy costs change equally for all miners. The real world is not that simple, but there is some correlation in energy costs across different regions as there is a global market for typical fuels (coal, natural gas, uranium, etc.).

It is also important to remember that transaction fees are not in proportion to transaction sizes. People doing large Bitcoin transactions could absorb much higher fees, so in all likelihood nobody would be priced out of the market (and Bitcoin would be dominated by large transactions, which is more or less the case right now).

I am not suggesting miners will react to the difficulty change. The difficulty change reacts to the miners.

Miners will shut their equipment down as soon as the electricity costs become greater than the rewards. When some of the miners shut their equipment down, that causes a difficulty change which makes mining profitable again for the remaining miners, and also keeps the transaction rate the same. The miners have no way to impact what transaction fees are acceptable because they have no control over the transaction rate.