It has been a day or two, right? (Former Minnesotan, now living in Hong Kong, so slightly out of it, but familiar with -20F/-30C and occasionally -40F/-40C.)

I think enough air circulation to avoid your breath causing mildew everywhere will mean without a ton of thermal mass, your house is going to get within a few degrees of the daily high within a day or two.