As I could not find any pre-existing good guidance on solving this, I ended up implementing it by having the resource servers (i.e. APIs) check the token using token introspection endpoint at the auth server (not the RFC version of introspection, just private one for now but I may add the RFC compliant version as well so it could be called by 3rd parties as we have not yet rolled out this OAuth2 based setup for our customers). Part of that return information from the introspection endpoint is this permission mapping between logical entities and scopes, so the resource server can know to which entities caller has which scopes. Of course, as our access tokens are not JWT and just opaque (encrypted) data, the introspection endpoint then need to decrypt it - just so it can quickly validate expiration etc. - and if valid, fetch a matching copy from the database that has more contextual information than what's encoded inside the client-supplied access token. Otherwise, the length would be prohibitive if token would be fully self-contained with these extra information.
Maybe this same kind of method could be used for your use case as well, encoding authorization policy adjacent to the access token in your backend, and looked up by app using it after it has received the token?
But the application currently has it's own authorization policy in a flat file which organizes users into groups and gives groups permission to take certain actions, a simple RBAC. I'd like to refactor this so that the authorization is delegated to the OAuth server.
Ideally I want to be able to ask a keycloak/okta server "now that user X is authenticated, are they part of the group Y?".
Since this isn't a common use case and everything is so abstract it's been very difficult to find an obvious path forward.