The access tokens go everywhere, to all services and are much more likely to be accidentally leaked / misappropriated. Refresh tokens are only used when talking (infrequently) to the access token minting service, so the scope of use is much much narrower.