For example, agile was never going to work according to the rules, because "individuals and interactions over processes and tools" are not how most companies work.
There's probably a parallel to draw with programming languages. A few people will use Haskell or Lisp or Idris and be at the bleeding edge, but most people are satisfied with Java that sometimes cherrypicks good features.
Both of those seem to be rooted in aversion to risk, or (inclusive) managers making most of the decisions. I'd say that's because developers tend to have different values from other people, often valuing stability a bit less and "the right solution" a bit more.
You're hinting at what the original DevOps idea was trying to solve. The traditional view is that Developers value shipping features while Ops engineers value stability/uptime. It should be easy to see why these differing values cause friction between these two groups.
The way DevOps was trying to solve this was getting _management_ to assign incentives/values/goals that would apply to _ both_ groups. An example might be increasing user engagement.
Aligning these two groups should be the main takeaway. The reason why The Phoenix Project book was so successful was it perfectly described this adversarial relationship. I'd certainly seen the same thing playing out at several companies I'd worked for.
First, elite practitioners identify flaws in the standard practices of some corner of the industry and figure out a way to ameliorate them. They give it a name so they can talk about it, write about it, organize conferences etc. They're mainly interested in disseminating this good idea.
The idea becomes popular, and a cottage industry springs up. Some of the original proponents become consultants, but there are others who were early adopters, and many more who are jumping on the trend. You also get an ecosystem of open source and commercial tools. This starts to warp the original ideas, as they stop being about purely about improving results and incorporate a bias for generating more consulting or tool revenue.
This drives wider adoption. Companies start looking for people with experience in the new thing, and people start putting it on their resume. It gets applied in situations it wasn't meant for, sometimes on purpose and sometimes by accident. You start to see "no true Scotsman" discussions where detractors claim they tried the thing and it didn't work and proponents claim they weren't doing it right. There's also cargo culting, where people who don't really understand the ideas try to implement the practices with varying degrees of success.
Eventually things get watered down to the point where the terminology starts to just mean "good" or "cool". The actual practitioners stop using it because it's no longer useful for communication and it falls out of favour.
But. All is not lost! The original ideas were good, and they've been widely disseminated. They get taught to young people joining the industry, and they start to be part of "how we do things." They don't need special names to distinguish them from "mainstream" practices because they are the mainstream. In fact, young people sneer at them because duh, without knowing what they were a reaction to.
You can see this with Agile, Lean and DevOps. On the business side, "disruption". The cycles are slower and maybe less intense in more CS and pure coding domains, but you find it there too with scripting languages, object oriented languages, TDD. Functional programming is leaving the consulting phase and seeing wider adoption.
This is probably not how we'd like to think things work. But they do work.
Royce: "These are the flaws in the 'Waterfall model'" -> everybody looks at his diagram and uses it [1]
Agile: "Individuals and interactions over processes and tools" -> SAFe [2]
DevOps: "Assign development and operations the same business goals so they work together, rather than against each other" -> "DevOps was a cultural shift giving development teams more control over shipping code to production" (this article)
[1]: https://en.wikipedia.org/wiki/Waterfall_model#History
[2]: https://scaledagile.com/safe-50/