I've been a part of many for-real security audits. This is largely incorrect scare-mongering. Separate admin workstations are symptomatic of a bad security posture and check-box security, they are absolutely not necessitated by nebulous concerns about the directory being readable.

Of course a server process which is designed to modify (among other things) group memberships needs different permissions than a user, why would that not be the case...?

If you don't like it being highly privileged, don't grant it the permissions. Or hire someone who can.