Buildings generally lose or gain heat throughout the day, proportional to the indoor-outdoor temperature differential. So a building that maintains livable temperatures for say, 12h a day, will still have those energy leaks when the owner is out, and they would have to compensate in the evening the exact amount of energy lost or gained to get back to comfortable temperatures.

So unless your home has such ridiculously bad insulation that it quickly approaches outside temperature after turning off heating/cooling, you won't see significant energy savings by going outside of the house half the time. Never mind multiple occupants, kids and the elderly, pets, diverging working hours etc.