Have you seen the wars and climate change going on right now? You're right to be pessimistic. This light come meme is such bs

There's a "system trap" where building intelligent systems or augmenting humans will grant at least a short term advantage, it has natural "arms race" dynamics.

Historically people have not done well at avoiding systems traps and maintaining any sort of global consensus not to do things. Particularly things that a) don't seem to be an immediate atrocity and b) grant economic advantage.