IIRC, I think first read the phrase "complexity catastrophe" in the book Design Rules: The Power of Modularity. Per the authors, it's when a system becomes so complex (interdependent) that the cost of any further changes far outweigh the expected benefit.
https://www.amazon.com/Design-Rules-Vol-Power-Modularity/dp/...
In The Matrix movies that is sort of a key premise of their world setting. There are words like reset, reboot, reload for it.
But I don't think it's a given, not all complex systems are prone to collapse; some systems, once past a given point, may even perpetuate their complexity practically indefinitely.
Consider life on Earth: even if a cataclysm occurs that results in mass extinctions, new life will eventually take over, in millions if not "mere" thousands or hundreds of years.
Until Earth itself no longer exists that is. Same goes for galaxies and the Universe at large.
At least trend-wise, it’d probably look similiar to a malthusian catastrophe (population reaches environment carrying capacity at time n, and then (greatly) overshoots it at time n+1, resulting in mass die off due to starvation, taking the population well below carrying capacity)
That said, it appears to me that as soon as we start to tread against entropy we might notice this phenomenon cropping up. Life seems to “collapse” on new designs and human understanding too seems to collapse, but in both cases it seems that only the system complexity collapses, not the entity complexity. I think that the collapse of the complexity of the surrounding system is the result of outside stimulus causing additional entity complexity to be added to the total complexity. In a stable, complex, system this has the potential for massive system destabilization, but it is not certain and it doesn’t seem to be the nature of complex system themselves to inherently collapse on less complex states, rather it is the result of stimulus. Moreover, if we look at total complexity, does it really become less complex or does the complexity just shift from the system to the entity?