1 is just plain wrong, AI models optimize human designed, mathematically defined loss functions, not arbitrary handwave-y incentives. We know exactly what these systems are trained to do.
Re:7 Weeks AI Alignment Curriculum
by mitthrowaway2 on Sunday March 05, 2023 @02:41PM
(#35033143)
One of the main reasons why AI alignment is a hard problem is that it's not easy to predict what behavior will result from optimization applied to a known loss function.
It is, the model will behave in a way to minimize (or at least approximate, most aren't convex) the loss function. This isn't magical or unexpected.
Re:7 Weeks AI Alignment Curriculum
by mitthrowaway2 on Sunday March 05, 2023 @02:53PM
(#35033261)
Yes, but when there are many degrees of freedom, that minimization might not happen in the way you expected, and if your loss function was only an approximation to your utility (because you weren't sure how to precisely codify "goodness"), the difference between the behavior you hoped for and the behavior you actually get might be very bad. For example, you attempt train an AI to get good behavior via a penalty button and reward button; an approximately-optimal strategy might include shooting the researcher to prevent him from pressing the penalty button, and then mashing the reward button itself.
I recommend actually watching some videos on the topic before insisting that the problem does not exist: https://www.youtube.com/watch?v=zkbPdEHEyEI
> and according to #1, we will always get it wrong
This is a point I've made. We are playing with paradoxes that proponents tend to ignore. They won't be solvable. The more sophisticated AI becomes, the more non-deterministic behavior arises. The emergent deception has been very interesting.
https://bounded-regret.ghost.io/emergent-deception-optimizat...
I've elaborated on #2 much further here as well - https://dakara.substack.com/p/ai-singularity-the-hubris-trap
1. AI systems respond to incentives, not commands, including incentives you did not intend to be there.
2. The technology will eventually outsmart its creators, making the imposition of reactionary controls difficult or impossible.
#1 was a good bet, and has already been proven with existing ML systems. In fact, it's one of the most common training blunders you can do and there are countless examples of artificial stupidity coming about from poorly thought-out incentives. AI are ruthless optimizers that will find the loopholes in your systems and maximally exploit them[0].
#2 implies significant harm if we get it wrong (and according to #1, we will always get it wrong). It is very plausible that AI will exceed human capability in at least some tasks, since there are already problem domains where AI dominates (e.g. chess, Go, etc).
My main objection to #2 is that the idea of a new structure or machine that exceeds human intelligence is not novel. Humans evolved to live in small tribes of no more than 100 people and we have not exceeded that limitation. So we're already several nested singularities deep[1] and humanity has not yet been destroyed.
On the other hand, humanity itself is a singularity: the system that created us (genetic evolution) is not capable of effectively controlling us anymore. We control it, and have been doing so since the dawn of agriculture. And we can see the harms that we have done to nature in pursuit of our own goals. So it's not irrational to extrapolate and see how an AI with more intelligence than us can do more harm to us.
[0] So are human children, but they do not have the I/O capability of a data center full of A100s.
[1] Agriculture beget civilization, which beget nation-states, which beget corporations, which beget computers. Each one is a singularity unto it's creators.