If that was your takeaway, then you missed the point. It was not an argument that it was a hard problem. It is precisely the argument that it is impossible as paradoxes are not solvable.
> As a matter of fact humans can do things perfectly, your CPU is executing billions of instructions perfectly every second
Yes, if you constrain the context small enough you get get the result you want. However, the concept of a ASI with free agency expands the context to a scope that we can't even reason about.
Similarly I'm not convinced that the scope is broad enough that it really cannot be reasoned about at all, either in its entirety or by breaking it up into smaller components (as we do for large problems already). I think you'd have to give some sort of impossibility proof on what about the problem is possible to reason about and what is not possible to reason about to convince me. (Note: chaos theory won't necessarily help you here, because probabilistic reasoning exists as well as do() operations by us, and if it must be used the argument must explain why certain catastrophic errors cannot be modeled and driven to arbitrarily low values.)
You might have some confused notions about free agency that if cleared up would resolve your other questions. This might be useful: https://wiki.lesswrong.com/wiki/Free_will_%28solution%29
It is an isolated narrow context. The problem with ASI is there is no such limitations. I addressed this point in the post, but another example would be the Kobayashi Maru Test.
Or rather, there has been no system devised that humans have not found a vulnerability. Why would AGI be any different? Alignment/containment must also address the possibility of human exploit of the system as well. So we are back at the beginning in either case. Which means a human could still devise a trap for the AI to leverage its ability or modify its alignment.
> Similarly I'm not convinced that the scope is broad enough that it really cannot be reasoned about at all, either in its entirety or by breaking it up into smaller components
We are already having challenges in this area. Example...
“The size and complexity of deep learning models, particularly language models, have increased to the point where even the creators have difficulty comprehending why their models make specific predictions. This lack of interpretability is a major concern, particularly in situations where individuals want to understand the reasoning behind a model’s output”
Yes, there's "difficulty comprehending" current deep learning models, put modestly. There's also interesting work in interpretability/explainability going on to work on that. Nothing there says "impossible". However, even granting that it is actually impossible to explain and reason about current AI methods to the degree we need to in order to create chains of logic for alignment purposes, that just says that it's impossible to reason about an AGI produced using such methods, if it's even possible to produce an AGI that way. It says nothing about it being impossible to reason about an AGI produced using different methods.
I understand you don't perceive the paradox. Nonetheless, my point is rather that when a paradox presents an impossible problem, then something about our premise is not understood. Or we need to find an entirely different concept to solve our problems. For example, an entirely different concept of intelligence not modelled strictly on how we perceive our brains. In summary, I think we are trying to solve the wrong problem. I might write something more depth at some point on that to explain my views.
> Incidentally there are many secure systems and schemes that have not fallen. But you'd probably write them off as too narrow for some reason.
This is precisely my point. We can build what is logically infallible to only be subverted by different means. If you can't hack the system, trick a human to give you access. This is how every such system has been defeated.
1. Congrats on recognizing that it's a hard (perhaps impossibly so, at least with the time left) problem! Too many (not any of the alignment people) have thought it's a trivial matter, or that Friendliness (the old word) will just "emerge", or that somehow being "superintelligent" will cause it to just smoke weed and play games all day like some super smart dorm room friend so there's nothing to worry about, or...
2. The idea of Friendliness or Aligned values is a difficult one purely from definitions alone, yes, let alone implementation, so that's part of what makes it such a hard problem. The idea of an AI tiling the solar system with smiley faces was once a response to a serious suggestion that one can just train the AI to optimize "human happiness". Imagine trying to formally model something like "If AI-Gandhi was offered a pill that made AI-Gandhi 2x smarter but also want to destroy the world, we know AI-Gandhi would not take such a pill because [...]" What's the [...]? A proof about its values? For actual Gandhi, we might as humans say "because he's Gandhi, and Gandhi does not want to destroy the world, and does not want to want to destroy the world". On the topic of the submission, I hope that any who take the course can by the end demonstrate many issues with things like the old and obsolete Coherent Extrapolated Volition proposal.
3. Containment is of course folly, you can look up the old AI Box arguments if you wanted more on that from back in the day. Hence the focus on Friendly (and more modernly, Aligned) AI.
4. I, an approximately +1 std dev IQ human, fully expect that it'd be no challenge for a -1 std dev IQ human to dig a pitfall trap that could not only contain me but kill me. Because gravity still applies even if I'm smarter than my adversary. Sure, an AGI will figure things out about physics that we don't know, and math too, but physics will still apply, I would bet that even a superintelligent AGI will never find a way to break the speed of light and travel the universe. But perhaps we're very wrong about special relativity. We'd have to be very wrong about everything if our proofs from first order logic are somehow awry. But that too highlights the level of challenge to the problem: we have to prove the AI is Friendly, with the reliability of first order logic. As a matter of fact humans can do things perfectly, your CPU is executing billions of instructions perfectly every second. (Or to be fair, there are errors, but those are modeled accurately and through correction and redundancy (with multiple systems for things that go into space) we can drive the probability of a total failure as low as we please.)