Well, a candle isn't enough to be a concern. But state of the art air tight passive houses include ventilator units that exchange air while retaining (or rejecting) as much heat as possible.