GitHub access tokens are a bit of a nightmare since you can't limit the permissions for a token. Only workaround I've found is to create another GitHub user for an access token and restrict that user's access.