That doesn't mean literally paying off the low interest loan though.

You can save up so that you have enough money invested to cover all the remaining loan payments and consider that having paid off the house. But don't send the money to the mortgage company, invest it into something stable that pays more than the mortgage interest and you're better off.

(Not historically always possible, but with today's interest rates pretty easy.)