Difficulty cannot scale downwards to the level you describe because then it ceases to be Bitcoin. It's supposed to be extremely difficult, this is what secures digital scarcity.

It doesn't help to speak in simplistic absolutes. Miners can definitely become more green and rapidly so. The massive move from China to the US that took place over the last few months likely has dramatically changed the energy mix to mine Bitcoin, for the better. There's also a Bitcoin mining council to promote best practices in renewable mining. There can be additional incentive structures (CO2 tax) to further discourage dirty mining.