Source: lived in a ~100 year old house in the Midwest for 15 years or so. The only place we got mold that I remember was inside the air conditioning unit.
You need a certain amount of wet for mold and mildew. Push the levers too far and you just end up with termites, or a failed foundation, and the whole thing gets replaced with something new.
The houses that have beaten their nemesis, water, are the ones most of us are willing to walk into more than twice.
Interesting whether this would happen in an entirely unheated building.
But good building design following current standards isn't going to be affected by this.
It's not just that spores work better in certain weather conditions, it's also that there are often more spores in those conditions as well. Mushrooms coming up after a rain, gambling that there will be another one in a couple days, for instance.