For areas where its not 90+ out for 8+ months a year, a large part of the comfort of the newer systems (and their larger SEER ratings) is that they can run at 1/2 speed (or whatever) to keep the AC unit running longer. Plus they are frequently sold with duct work upgrades.
This is also why a "properly sized" system with a high EER can frequently save money over a high SEER system with a lower EER in FL/TX/LA/MS/etc.
The problem though is that pretty much 100% of AC companies are just going to put in the same size system as they took out, and the home builders are just seat of the pants overspecing everything. It should be possible to determine heat load, and TXV flow/etc over a year, and then come back and resize the condenser/evaporator/airflow to go just slightly over the worse case. But no one is going to do that, and if they mess up and the customer can't keep the house at 78F when its 110 out, its going to be a costly problem to fix. So, better to just overbuild everything and deal with the significant efficiency decrease.
A lot of builders in humid climates will install AC and central dehumidification, which can work better. In theory, one can tweak the indoor heat exchanger temperature to change the sensible vs latent cooling ratio, and there is at least one obscure heat pump vendor that claims to do this on the fly. I don’t know how well it works in practice.