My understanding, and it may be wrong, is that if an attack walks off with your data, then wouldn't be able to refresh the access token you have on hand without the client ID and secret and so the extent to which they could compromise the authorized accounts is limited in time.