This whole thing makes me think of that short story "The Machine Stops".
As we keep burrowing deeper and deeper into an overly complex system that allows people to get into parts of it without understanding the whole, we are edging closer to a situation where no one is left who can actually reason about the system and it starts to deteriorate beyond repair until it suddenly collapses.
We are so, so far beyond that point already. The complexity of the world economy is beyond any one mind to fully comprehend. The microcosm of building black-box LLMs that perform feats we don't understand is yet another instance of us building systems which may forever be beyond human understanding.
How is any human meant to understand a billion lines of code in a single codebase? How is any human meant to understand a world where there are potentially trillions of lines of code operating?
Wow, I've been there ! Years ago we dragged a GIS system kicking and screaming from its nascent era of a dozen ultrasharp dudes with the whole national fiber optics network in their head full of clever optimizations, to three thousand mostly clueless users churning out industrial scale spaghetti... The old hands wanted a dumb fast tool that does their bidding - they hated the slower wizard-assisted handholding, that turned out to be essential to the new population's productivity.
Command line vs. GUI again... Expressivity vs. discoverability, all the choices vs. don't make me think. Know your users !