One of the issues is that your house heats/cools with a certain thermal inertia, so at the end of the day it's almost as hot as it gets (as it cools during the night and heats through the day). In the morning it's as cold as it gets, in the sunset it's as hot as it gets

There are probably some biological factors (still, I feel colder when sleeping, but of course I have a bad time sleeping when it's hot)