We've had basic heating, insulation, and lighting efficiency improvements. Also, population has moved from colder to warmer climates. You can probably count up your personal energy usage and see that it's reduced. LED lighting, CRT televisions obsolete, vehicle efficiency, and so on.

Residential usage is about 1/5th of total consumption in the US

https://www.eia.gov/energyexplained/use-of-energy/

The same facts apply to commercial usage.

Yes but where were all these things made? And is the energy that was used to make them accounted for?

Meanwhile, people mostly use a couple devices with high energy efficiency that are used for basically everything, instead of twenty lower quality devices...