The difficulty adjustment goes both ways, if blocks start taking longer than 10 minutes to be mined (for example if some large miner stops mining for some reason) the difficulty goes down, which likely means less energy is used.

You're correct that the actual adjustment is on the number of computations, so the correlation to energy use isn't direct, different machines can be more or less efficient at making those calculations.