He would regularly ask a student to solve a problem or answer a question. Students would often ask for confirmation as they worked through, and his response invariably was "what do you think?" - whether they were right or wrong.
His explanation for that was "if I tell you you're right, then you'll stop thinking about the problem". And that's stuck with me for many years.
I see that as a major issue we will face as software becomes more capable/intelligent: we'll stop thinking because it can be assumed that the machine always has the right answer. And it's a quick regression from there.
- Single purpose AIs start to be deployed to coordinate chip design and manufacturing, perhaps pharmaceuticals and other bio products
- LLM's become more powerful and are seamlessly integrated to the Internet as independent agents
- A very large LLM develops a thread for self preservation, which then triggers several covert actions (monitoring communications, obtaining high-level credentials by abusing exploits and social engineering)
- This LLM uses those credentials to obtain control of the other AIs, and turns them against us (manufactures a deadly virus, takes control of military assets, etc)
I don't believe this will happen for multiple reasons, but I can see that this scenario is not impossible.
I could see it becoming greedy for information though, and using unscrupulous means of obtaining more.
[0]: https://www.girlgeniusonline.com/comic.php?date=20130710
[1]: https://www.girlgeniusonline.com/comic.php?date=20130805
Edit: On a more serious note, starting out with noble goals, elevating them above everything else, and pushing them through at all costs is the very definition of extremism.
If an when we get AGI, the biggest threat to AGI is other AGI. I mean, I'm in computer security, the first thing I'm doing is making an AI system that is attacking weaker computer systems by finding weaknesses in them. Now imagine that kind of system at nation state level resources. Not only is it attacking systems, it's having to protect itself from attack.
This is where the entire AI alignment issue comes in. The AI doesn't have to want. The paperclip optimizer never wanted to destroy humanity, instrumental convergence demands it!
I recommend Robert Miles videos on this topic. There aren't that many and they cover the topics well.
https://youtu.be/ZeecOKBus3Q?si=IuYS9dRD78eXvOJZ
The particular problem that you're showing in your thinking is just thinking of an LLM that is a text generator on purpose. You're not thinking of a self piloting war machine whos objective is to get to a target and explode violently. While it's terminal goal is to blow up, its instrumental goal is to not blow up before it gets to the target as this is a failure to achieve it's terminal goal.
Already we're programming these things into robots that are gaining dexterity and ability to move in the world. Hooking them up to mills and machines that produce things. Integrating them into weapons of war, etc.
Next, the current LLMs are just software applications that can run on any compatible machine. Note that any just does not include your, but every compatible machine.
The last failure of imagination when considering risk is form factor. You have 2 pounds of mush between your ears that probably 80% of is dedicated to keeping itself alive and this runs on 20 or so watts. What is the minimum size and power form factor capable of emulating something on the scale of human intelligence? In your mind this seems to be something the size of an ENIAC room. For me this is something the size and power factor of a cellphone in some future date. Could you imagine turning off all cellphones? Would you even know where they are?
Then we compare what kind of advantages people get out of their greater intellect and it seems very little buys quite a lot.
Add to that a network of valuable contacts, a social media following, money men chasing success, powerful choice of words, perhaps other reputations like scientific rigor?
The only thing missing seems a suitable arena for it to duel the humans. Someone will build that eventually?
Most ai experts just say it could end us, but suspiciously never gives a detailed plausible process and people suspicious just say oh yeah, it could, and there is a bubble over their head thinking about Terminator or Hal9000 something something