Fireplaces typically draw air from the house to sustain the fire, then up the chimney or out a vent. That creates negative pressure, which draws cold, unconditioned air from outside into the house from wherever air can leak in.

How this works out in terms of net heat gained/lost will depend on the fireplace design, but, from what I hear, it's not uncommon for this to be a net-loss type of situation.