Great (and terrifying) point. In that light it would seem that the major blocker for these types of engineering projects isn’t technological advancement as much as advancement in governance/social organization/psychology.
Let's not forget about the crazy either. Both those with mental illnesses making them destructive, and those mutilated by the system. Think of the occasional random people going on shooting sprees or bombing runs. Imagine them with access to a home pathogen lab, in 20 years.
The major blocker full-stop. Adorno once used the V2 flying bomb as a metaphor for our society: as the marriage of total technical brilliance with utter futility and idiocy. I think it's frankly surprising that we didn't blow ourselves up in the cold war. We're extremely developed as technical thinkers, but as social thinkers, half of us vote Trump, and the other half don't have the words to express why that's awful.
A civilization with the kind of industrial base and know-how to build Dyson swarms probably also has the ability to create smallish self-sustaining colonies, in space or on/within uninhabitable planets. With enough of those floating around, even if most of civilization was wiped out, it seems likely that someone, somewhere would survive... and potentially be able to reseed a future civilization. So there’s hope.
Worse, in such a civilization, it's pretty likely that even small organizations would have the kind of tech you'd need to induce a nuclear winter on a given planet. After all, you just have to crash an asteroid on the right spot, then you have a mass extinction that human civilization would definitely not survive.