The biggest problem though is the dryness during the winter, cold air comes in and expands, lowering the relative humidity, warm moisturous air is pushed out by mechanical fans. You get extreme dryness, lower then in the desert, which is not good for mucus membranes. I believe that if we could revolutionise air conditioning systems we could prevent the next pandemic (and maybe help the global climate).
Most people these days want more than natural cooling. As soon as you decide to have an AC system installed more insulation is better for the reasons you state.
I'd think barns would have very low mass in comparison to their size, but would stay close to surrounding temp by way of having very little insulation and being very "drafty"
It doesn't completely shift the issue to later in the season. Yes, it will shift some of it, but you still have the night/day average effect. This can still be beneficial, especially if in a shady location.
Barns can be drafty if you are talking about wooden barns. Stone barns, or stone foundation barns built into a hillside, can actually be cooler inside than outside during the heat of the day. That's part of why horse owner sometimes leave their animals in the barn during the day and turn them out at night (typical summer schedule).
Of course, once the heat is in, you need a way to get it out (at night, through windows).
Unlike a greenhouse, insulation is not directional.