> There are costs, as heat or/AC will have to run more to keep the temperature in the right range.

Heat recovery ventilators can largely solve this problem, while also reducing particulate pollution indoors.