Construction makes a huge difference -- especially airflow. Whoever made my current apartment was a genius because in the winter, the south side gets sun through the windows for about 6 hours, while in the summer there is no direct sunlight in any window. My previous apartment was an oven.
North American attitudes to seasonal temperatures are different than where I live now. In the winter here, the lows here are near freezing (1-2C), and we always have a few days in the summer well over 40C (105F) (luckily once it gets over 35, the humidity drops quite a bit). It's quite normal here not to heat in the winter, nor cool in the summer. Experiencing the seasons is considered to be "culturally enlightening" ;-) And joking aside, I've found that I like this way of living. Like you, I would never have thought it possible before I tried it, though.
Having said that, you can't always do it. I need some heat in the winter these days, due to health conditions. I worry about my mother in law who, at 80, usually refuses to run her AC in the summer. You do need to be careful of your health. You also need to be sensitive to the fact that architects build for the prevailing culture. Where I live, they work very hard to deal with this kind of thing. Where you live, they will probably optimise for something else.
Like I said, it's a very cultural thing -- and that's the beauty of the world. There are different cultures and values, each with their own charms. It's totally fair to say that you like the way you live now (and more power to you!) But it's not impossible (nor bad) to live differently.
We've got the technology now to evolve this even further by doing computer simulations to come up with better designs, and by techniques like 3D house printing with concrete to make intricate ventilation systems that can circulate air through the floors and up through walls with zero energy cost.
A thousand years ago you had to build things this way, you had no alternative, but now it's taken for granted that air-conditioning will solve all problems.
No, it isn't. I have an enormous oak tree that overhangs the southern roof. My windows have a very low level of solar radiation they allow through. The ambient air temp at night is almost 80F. The house is built of concrete block, and stores heat during the day, releasing it at night. We're routinely at 60-80% humidity.
Just waiting for some solar shingles to be available (hurry up Elon!). Don't much care how long I run the AC every day as long as I net out 0 with solar generation.