Ground and water source source heat pumps can keep their efficiency down to extreme temperatures as their source maintains its temperature. Air sourced heat pumps can go down to -20C and still be above 100%,(though this is notably below their peak in warmer ambient temperatures) and they often have fallback resistive heaters that are independant of outside temp and are 100% efficient (though probably not sized to work alone generally).
So whether they are efficient and expensive really needs a complex comparison including up front investment in digging trenches for geo or pipes, ongoing costs of fuel, predictions of heating and cooling days for the region, whether insulation makes more sense than more heating and so on. I don't think there's anything that strictly rules out heat pumps in most places people live or restricts them to California.
Stockholm getting heat from the North sea is a good example of their counterintuitive potential in colder climates.
A heat pump being "efficient" can casually be thought of as "multiples better than just resistant heating" (but is measured in cooling and heating efficiency, seasonal performance, etc.)
This drives whether the heat pump is economical or not, because the more heat you have to produce directly from electricity, the less economical your installation is, compared to other heat sources. A gas vendor in Washington, DC area estimates annual savings of $1,500+ by using natural gas heating and water heating over electric, including air-source heat pumps, and significantly lower lifetime costs: https://html5.dcatalog.com/?docid=2db77d07-d752-43b8-bfaa-a8...
You say $1500 saving but that's against resistive heat, not heat pumps, which are only 150 to 350. Which could be small enough a difference that providing cooling with one system may cover it.
And those are only operating costs, and I'd guess if you didn't have a gas grid magically installed for no cost, then propane or oil are the two options you'd more realistically go for rather than pay build a gas connection, whereas a house without an electric grid connection is pretty rare and could be considered a sunk cost in most cases.
Obviously for many existing homes you do magically have gas pipelines for free, but in the context of new builds, you'd want to compare those costs.
A similar argument applies to electric cars. Electric motors are so much more efficient that they still beat the alternative after being transmitted and stored in a battery even if the original source is burning gasoline to generate electricity.
That's before getting into carbon and pollution.