Depends on house type. For american-style wooden houses that might be true, but where buildings are built primarily using concrete with outer insulation, their thermal mass and inertia is much bigger.