I could see it becoming greedy for information though, and using unscrupulous means of obtaining more.
[0]: https://www.girlgeniusonline.com/comic.php?date=20130710
[1]: https://www.girlgeniusonline.com/comic.php?date=20130805
Edit: On a more serious note, starting out with noble goals, elevating them above everything else, and pushing them through at all costs is the very definition of extremism.
If an when we get AGI, the biggest threat to AGI is other AGI. I mean, I'm in computer security, the first thing I'm doing is making an AI system that is attacking weaker computer systems by finding weaknesses in them. Now imagine that kind of system at nation state level resources. Not only is it attacking systems, it's having to protect itself from attack.
This is where the entire AI alignment issue comes in. The AI doesn't have to want. The paperclip optimizer never wanted to destroy humanity, instrumental convergence demands it!
I recommend Robert Miles videos on this topic. There aren't that many and they cover the topics well.
https://youtu.be/ZeecOKBus3Q?si=IuYS9dRD78eXvOJZ
The particular problem that you're showing in your thinking is just thinking of an LLM that is a text generator on purpose. You're not thinking of a self piloting war machine whos objective is to get to a target and explode violently. While it's terminal goal is to blow up, its instrumental goal is to not blow up before it gets to the target as this is a failure to achieve it's terminal goal.
- Single purpose AIs start to be deployed to coordinate chip design and manufacturing, perhaps pharmaceuticals and other bio products
- LLM's become more powerful and are seamlessly integrated to the Internet as independent agents
- A very large LLM develops a thread for self preservation, which then triggers several covert actions (monitoring communications, obtaining high-level credentials by abusing exploits and social engineering)
- This LLM uses those credentials to obtain control of the other AIs, and turns them against us (manufactures a deadly virus, takes control of military assets, etc)
I don't believe this will happen for multiple reasons, but I can see that this scenario is not impossible.