I use this feature all the time, and never have to auth on my phone... Do you have some higher security setting than default, like requiring passcode and not allowing biometric unlock?