Unless you live in an adobe/stone/brick house, the thermal mass of your walls is pretty small. Traditional plaster would add a limited amount to a stud-framed house, but in most US houses, what you've said isn't true.