The difficulty and energy cost is a function of price, not the other way around. If the price goes down, miners aren't profitable and some have to quit, which makes the difficulty adjust downwards.