If you're counting that, the efficiencies of using all the electricity to communicate and heat/cool your home because you're in it also matter

Well, I was responding to a comment only about commuting, but yes. Housing costs and the energy (including direct construction, for those who made it and profited from it, and maintenance/heat etc) used are enormous. It's 30-40% of most people's budget both $ and Carbon. If you're living in a $1M+ house to avoid $10k worth of annual commuting the environmental savings may not be worth it.

Well the home gets heated anyway. The fridge pump is pumping anyway.

Everyone is encouraged to get a certain number of steps in per day (some say 10,000) for health. If you don't get those steps walking to the office, you're going to have to get them some other way, like by walking somewhere else or, worse, using a treadmill that also consumes energy in the process.