And that’s when you realize that searching in AD is actually dog slow, and you are better off just syncing the whole thing to a proper database, then checking if the object still exists after.
Seriously, why does a search that takes 1ms in postgres take 3 full seconds in AD?
Because when specialization occurs, every specialized component (including people) is selected against performance and other non-essential metrics.
The 700k groups also comes about when the security tools are all inter-operating at the wrong abstraction level. If a third party appliance needs to import all 700k of your security groups it means the appliance is performing authorizations itself, logging it differently than your other apps, and even make decisions based on stale data it's cached (you can't load all 700k groups on every request.)
This task should really be delegated to a dedicated authz system, too bad more of the world doesn't run on Zanzibar.
I was able to find us an open-source / self-hosted solution that worked far better (bookstack). But I was amazed at how far the company selling the software got with us -- seems like most never get pushback, they just make sales by labeling themselves as fit-for-purpose regardless of whether or not they generally work.
Like the poster of the blog article, I too have worked in very large enterprise. Think 200K staff, 10K distinct servers, etc...
I had the lucky opportunity to assist the CTO in hundred-million-dollar enterprise software purchasing decisions. I got to interview vendor after vendor, grilling them to see if they're compatible with "large enterprise".
Most weren't.
This wasn't about some Kafkaesque hoops they had jump through while blindfolded and on fire, although I'm sure that's the impression most of them got.
The requirements were simple and perfectly sound architecture principles that they either met or did not meet. If they didn't meet them, then maybe their software was "good enough" for tiny clients, but would never work at scale.
I even made a list, which from memory was something like:
1. Support single-sign-on from external sources such as LDAP, OAuth, whatever. (We're not going to sync our directory and every user password to your insecure garbage software.)
2. Have some sort of audit log. Administrative or configuration changes especially. (Some places have hundreds of admins, not all of whom are fully trusted.)
3. Have an unattended installation process, even if it's just VM cloning or whatever. (I'm not clicking through your GUI wizard 500 times at 3am on Sunday morning.)
4. Allow incremental migrations/upgrades. I.e.: The "easy" process of stop-the-world; one-way-big-bang; start-the-world process is not viable in an enterprise with 10K tenants where the upgrade may or may not work for all of them on the first try.
5. Scale. This can be as simple as having indexes on "user" and "tenant" tables, which is easily overlooked when the typical dev works with a scale of 1 user and 1 tenant. Similarly, using combo boxes / drop downs is a no-go for most fields such as security groups. (We have 700K security groups. No, we can't delete 699.9K as a workaround to make your GUI not crash from an OOM error.)
6. Accessibility is mandatory. With up to 200K permanent and 1 million occasional users, every disability will be represented. Not just vision or hearing problems, but motor-neuron issues, amputees, etc... (You name it, we have someone on staff with it.)
Etc...
Seen in that light, Enterprise software starts to make sense. It's not baroque or malicious, it's just taken on a certain form to suit a purpose.
For example, I didn't understand why Active Directory uses search dialog boxes for every "picker" GUI control instead of a simpler drop-down or something similar... until I worked in an environment with 2 million objects in the directory.