Let's hookup Metasploit to AI, give it seed crypto totalling $100k USD, purchase access to buy 0d's from Zerodium, domain registrar, email, VOIP phone numbers, and let it loose on the world to see what happens. Oh, and if we want self-designing killer drones and self-designing software and systems, that's cool too because every other country is going to do it.

Like becoming a chemist instead of a pyro: I agree that things should be tried, but you don't want to light a segment of an Space Shuttle SRB motor segment to defrost your windshield every day. Both are playing with fucking fire, but the habit should be not normalizing a routine of unsafe practices. There is no "Manhattan Project" to be had here because there is no clear, unclassified achievement objective to produce a universal, static forever, turnkey "nuclear weapon".