Pets. Other family members. Plus the fact that you're "daytime" heating/cooling a home from 6:30-8:30 am and 3:30-10 pm anyway, and that modern or updated homes keep their temperature for several hours. So the debate is really over a few degrees for a couple of extra hours (Vs. heating an entire other building that may not otherwise exist).

I know at our workplace heating on Monday mornings has to start several hours earlier than on Tuesday because the office lost almost all of its built-up heat from the previous Friday. I also know that they continue to run the heating on holiday-Fridays/Mondays because if they don't by the following Monday/Tuesday, the temperature will drop too low and could damage equipment.