That is a good point, I think you're exactly right. I just tried to figure out how many kWh is equivalent to a therm of natural gas, and the answer of course is 'it depends.' In round numbers it seems like 1 therm is about 30 kWh.
Using myself as an example, during a cold month I might use about 90 therms. In a hot month I may use 1900 kWh for cooling. Gas is significantly cheaper indeed, though it's noticeably more energy (and that's after converted to hypothetical electricity, not the energy content of the raw gas).
What a PITA to compare that. And it's a pretty rough comparison, indeed, because gas is predominantly going for heat in the winter, but I have a gas stove and a gas water heater too. And I have an electric car, which pushes the kWh total up a bit. I'd have to do a much more detailed analysis if I actually wanted to compare the energy usage.
Because heat pumps are now so efficient, all the new apartment construction I see in NYC uses them. (While older buildings use dirty oil to heat water for radiators.)
My money is on "gas won't be cheaper for much longer, even where it is now" personally, though.