That only applies within the original loan.

You can just refinance out of it (when the numbers are right), and it’s often easy since you now have equity, a proven payment history, and are presumably looking at even lower payments on the new loan.