I've noticed that for sites I visit rarely there will be some sort of drift from the auth app and I'm often forced to use a backup code.

I'd rather just have a whole list of allowable factors and the opportunity to configure one or more of them depending on my personal risk tolerance.