We live in a world of asymmetric warfare in which attackers can screw up thousands of times, but defenders only need to screw up once. Dramatically increase the payoff of compromise, and the risk increases accordingly.
I know this smacks of security by obscurity, but in the real world, incentives for attackers are absolutely part of the risk calculation.
Another pitfall is when people centralize both authentication and authorization which leads to allot of issues mostly loss of granular control over permissions and information exposure.
If you are using some sort of SSO/Global Token based authentication you should let your applications handle the authorization, and you should not store information about the user's privileges within the token itself.
To me it sounds like recognizing a good solution to the problem and implementing it. Unless you have a good counter-reason to offer why a centralized authentication service is a bigger problem?
(Note there is a sliding scale on how often you need to hit the security service. You can have every single request going there, or you can have it give out tokens with varying permissions your services then can validate themselves. Cloudflare's blog recently described how they use internal CAs to authenticate services against each other, basically only requiring a service to once get a certificate, there are concepts like Google Macaroons, ...)