The correct way to handle this during a routine stop is to inform the officer that you do not answer questions. In the US, you are not required to answer any questions.

Also, the police have a limited time to conduct the stop. Once they give you back your license, you can immediately leave regardless of what the officer requests that you do. The stop is over at that point, and they cannot legally detain you any longer.