Ehh... but at least for Google's implementation, you won't have a refresh token in the first place if the user doesn't grant "offline access" on a special page that comes up after login.

In our usability testing, we've found that this freaks a lot of people out and reduces adoption. The benefits of basically outsourcing our session management to Google don't outweigh this... so we use JWT for auth only, and then use that token as a session key for our own local solution.