You want to control your ventilation. Making a loose house just means you get ventilation, but not on your terms. HRV/ervs in a tight house allows you to ventilate your house, how you want, where you want. A loose house might be ventilating your house through insulation filled with dust and rat poop. You will never know.

I can see daylight between my ceiling and exterior walls, so I know where the air cubes from in my house...

In an old house, you will know about the rat poop by the smell after you turn on that new AC unit.

In theory if you get just the right number of air exchangers such that the same amount of air moves through as it would with a 'loose' house, you get the same airflow with almost none of the convective heat loss/gain.

This is however, very much in theory, since as far as I can tell we're still in the early stages of figuring out how to do airtight houses well. My home was built in ~2006 and has an unhealthy CO2 build-up if I close the windows, since it has no active ventilation whatsoever.