Airborne allergens can be an issue, and dust more generally.

Seems like it would be more efficient to wear a climate controlled bubble suit than heating up entire rooms. And you could even go outside when it is cold or hot or dusty.

You do realize that “open the window and hear that air” is massively energy inefficient, right?

I don't know; is it more energy inefficient than having a fresh air intake via a duct? Maybe they are comparable in the absence of a heat recovery ventilation system (mentioned by a sibling comment upthread)?

You don't necessarily need to take in fresh air to freshen air in certain ways. It might suffice to pass it though a good filter.

HEPA and carbon filters do wonders for cooking smells, particulates etc, but unfortunately do nothing for the main reason you want air exchange with outside, which is CO2 buildup.

Our house has a heat exchanger that exchanges air. So the outgoing air changes the temperature of the incoming air.

You want to open windows in opposite sides of the house to quickly circulate air from outside to inside. 5 minutes should be enough. It will get colder, but most of the heat is in walls and furnitures and as long as you only open the windows for a few minutes it should be quick to heat up the air again.