As to the sunlight - while we don’t get any directly into the house for 2.5 months in the winter, the far side of the valley, which is practically barren, ends up brightly lit, bringing a lot of indirect light in.
Finally, for us, the summer is a killer - it can be over 40 for weeks on end, and the nights on the shadier side are much cooler, as we don’t have superheated rocks re-emitting IR, instead the canopy reflects and absorbs most of the solar irradiance. The daytime air will still be sweltering, but it doesn’t transfer heat anything like direct insolation. So the effect is more about the environment around the structure than the structure itself - although regarding the house the roof is exposed to the sky, as it’s heavily insulated and doesn’t matter, but the walls, which have a decent thermal mass being solid lumber, are always shaded, as are the windows, mostly through self-shading due to the orientation. Again, punishingly bright in there in the summer even without direct sunlight - most folks near us just eschew windows, or cover them with heavy shutters and so have no light and no view in the summer.
I still would always take that 4h of sunlight over 0h and just invest in better insulation. Don't you miss the sun in winter?
But if you don't, well, then it makes sense. North facing sites are usually quite cheaper as well.