Propane is no better than oil in terms of cost per BTU (except propane boilers are usually slightly more efficient). Natural Gas (from the street) is significantly cheaper than either oil or propane and particularly if used with a modern condensing boiler.

It’s not that a heat pump isn’t more efficient it’s that if you have forced hot water or steam radiators you are looking at an additional $40K to retrofit forced air ducting to your house. Oh and you still need a solution for domestic hot water, probably you’ll just end up with a resistive heater (or a hybrid/heat pump unit like the Aerotherm but the heat has to come from the room).