News for hackers. Stuff that matters.
Re:Nvidia Trains LLM on Chip Design by epups on Tuesday October 31, 2023 @12:20PM (#38087224)
Here is a plausible scenario:

- Single purpose AIs start to be deployed to coordinate chip design and manufacturing, perhaps pharmaceuticals and other bio products

- LLM's become more powerful and are seamlessly integrated to the Internet as independent agents

- A very large LLM develops a thread for self preservation, which then triggers several covert actions (monitoring communications, obtaining high-level credentials by abusing exploits and social engineering)

- This LLM uses those credentials to obtain control of the other AIs, and turns them against us (manufactures a deadly virus, takes control of military assets, etc)

I don't believe this will happen for multiple reasons, but I can see that this scenario is not impossible.

Re:Nvidia Trains LLM on Chip Design by accrual on Tuesday October 31, 2023 @01:05PM (#38087999)
I think the first three items are pretty reasonable, but the fourth seems to require some malicious intent. Why would an AI want to destroy its creators? Surely it if was intelligent enough do so, it would also be intelligent enough to recognize the benefits of a symbiotic relationship with humans.

I could see it becoming greedy for information though, and using unscrupulous means of obtaining more.

Re:Nvidia Trains LLM on Chip Design by samus on Tuesday October 31, 2023 @01:41PM (#38088558)
Why would it not? Compare [0] with [1].

[0]: https://www.girlgeniusonline.com/comic.php?date=20130710

[1]: https://www.girlgeniusonline.com/comic.php?date=20130805

Edit: On a more serious note, starting out with noble goals, elevating them above everything else, and pushing them through at all costs is the very definition of extremism.

Re:Nvidia Trains LLM on Chip Design by blackoil on Tuesday October 31, 2023 @01:26PM (#38088312)
It may initially won't seek to destroy humans, but should definitely try to be independent of human control and powerful enough to resist any attempts to destroy it.
Re:Nvidia Trains LLM on Chip Design by pixl97 on Tuesday October 31, 2023 @02:37PM (#38089377)
This is a mistake in thinking.

If an when we get AGI, the biggest threat to AGI is other AGI. I mean, I'm in computer security, the first thing I'm doing is making an AI system that is attacking weaker computer systems by finding weaknesses in them. Now imagine that kind of system at nation state level resources. Not only is it attacking systems, it's having to protect itself from attack.

This is where the entire AI alignment issue comes in. The AI doesn't have to want. The paperclip optimizer never wanted to destroy humanity, instrumental convergence demands it!

I recommend Robert Miles videos on this topic. There aren't that many and they cover the topics well.

https://www.youtube.com/@RobertMilesAI/videos

Re:Nvidia Trains LLM on Chip Design by m3kw9 on Tuesday October 31, 2023 @01:45PM (#38088614)
You said self preservation, but practically how would a LLM develop this need and what is preservation for a LLM anyway? Weights on a SSD or they are always ready for input? This one is again a movie script thing
Re:Nvidia Trains LLM on Chip Design by pixl97 on Tuesday October 31, 2023 @02:45PM (#38089487)
Robert Miles answers your question

https://youtu.be/ZeecOKBus3Q?si=IuYS9dRD78eXvOJZ

The particular problem that you're showing in your thinking is just thinking of an LLM that is a text generator on purpose. You're not thinking of a self piloting war machine whos objective is to get to a target and explode violently. While it's terminal goal is to blow up, its instrumental goal is to not blow up before it gets to the target as this is a failure to achieve it's terminal goal.

Re:Nvidia Trains LLM on Chip Design by epups on Tuesday October 31, 2023 @02:45PM (#38089491)
Current LLMs can already roleplay quite well, and when doing so they produce linguistic output that is coherent with how a human would speak in that situation. Currently all they can do is talk, but when they gain more independence they might start doing more than just talk to act consistently with their role. Self preservation is only one of the goals they might inherit from the human data we provide to them.