It depends on what you mean by "let it get cold" and how good your insulation is and a number of other factors.

But thermal mass can be a big thing. My house isn't even all that well insulated, and when it is 0º F outside, it only drops about 10 degrees if the furnace is off all day.