Groups are not granular enough. `pkaction` lists 315 different actions on the system where I just ran it. You'd need 315 groups to begin to reflect that level of granularity...

Groups are also not dynamic enough. When evaluating these rules, one of the inputs is whether the client requesting the action is a member of no session at all, or an inactive console session, or an active console session. You can't represent these with groups because you can't grant/revoke group membership to a process while it's running. (A process with a group membership can also stash away a setgid executable that can be used by the user to regain access to a group that they've been removed from later on, so it's not even possible to cast-iron-guarantee that access to a group has been revoked without inspecting the filesystem...)

How many of these groups are really needed ? Seems like overengineering.

I don't think you can have different group per session?

Sure you can, see pam_group.