This has been a lot of my career. All my long, successful jobs have lasted around 2-3 years, and it's the time it takes for the company to realize the initiative wasn't worth it. I count four of these stints where I left, and the team was essentially dissolved within a year. I'm not saying I was the key person for the project; I just saw the writing on the wall.
It's been career limiting not having a large-scope project on a growing team, but I don't know if this is something about me, something about the projects I'm a good fit for, or the reality that a lot of projects and teams don't survive 5 years, and I've been exposed to some survivor bias of smart people who have gotten lucky.
Another example is the changing nature of the contracts people are hired under. I'm a scientist, and a lot of positions are inherently term-limited. You end up with a lot of organizations with two different groups of employees: employers who have worked there for decades, and employers who are temporary and work there for a few years hoping to be converted to permanent. Just like in the situation you describe, if you end up on a project that just isn't working out, that does not bode well for your own prospects. That has nothing to do with your own capabilities or how hard you worked on the project. And just like you mention, a lot of the "permanent" people have survivorship bias and do not really understand that they ultimately got lucky and timed the market well.
And to be frank, layoffs are scary for any industry, because they usually are correlated with layoffs at other companies. A rising tide lifts all boats, but a sinking tide is difficult to escape. You might be laid off at a time when there is no other job to hop to.