Buildings generally lose or gain heat throughout the day, proportional to the indoor-outdoor temperature differential. So a building that maintains livable temperatures for say, 12h a day, will still have those energy leaks when the owner is out, and they would have to compensate in the evening the exact amount of energy lost or gained to get back to comfortable temperatures.

So unless your home has such ridiculously bad insulation that it quickly approaches outside temperature after turning off heating/cooling, you won't see significant energy savings by going outside of the house half the time. Never mind multiple occupants, kids and the elderly, pets, diverging working hours etc.

A home that has AC running half of the day will use a lot less energy in total than one running all day long. There are substantial efficiency losses from heat leaking in, which will be exacerbated by trying to maintain a larger temperature delta 24 hours of each day. A house is not a perfect temperature battery on the scale of hours, far from it.

Depends on house type. For american-style wooden houses that might be true, but where buildings are built primarily using concrete with outer insulation, their thermal mass and inertia is much bigger.

Maybe the solution is not having the AC running the whole time? I mean why do you need your house cooled to 19 degC?

It's going to vary entirely depending on your dwelling type. But that's a red herring I think. We need to make homes energy efficient anyway (regardless where people work) and once you've insulated/moved to more efficient heating/cooling then the cost of keeping your dwelling running for the hours you're personally out should be pretty low. Not to mention however many households have to be kept warm/cool because there are other household members home.