The big idea is diminishing marginal returns – increasing societal complexity pays off enormously at first, but eventually the returns level off while the overhead remains high or even continues to increase. Collapse is then seen as a natural (and not necessarily cataclysmic) response to this. It's a process of simplification.

Tainter also points out that collapse may not occur (1) when there are strong, neighboring powers, which simply absorb the foundering society, or (2) when there are strong cultural reasons, such as national identity, for which the members of the society may be willing to put up with the bureaucratic overhead indefinitely (in order to prevent being absorbed); he gives Europe as an example, I believe.