> I think my example of a pitfall trap in 4 is sufficient to demonstrate that

It is an isolated narrow context. The problem with ASI is there is no such limitations. I addressed this point in the post, but another example would be the Kobayashi Maru Test.

Or rather, there has been no system devised that humans have not found a vulnerability. Why would AGI be any different? Alignment/containment must also address the possibility of human exploit of the system as well. So we are back at the beginning in either case. Which means a human could still devise a trap for the AI to leverage its ability or modify its alignment.

> Similarly I'm not convinced that the scope is broad enough that it really cannot be reasoned about at all, either in its entirety or by breaking it up into smaller components

We are already having challenges in this area. Example...

“The size and complexity of deep learning models, particularly language models, have increased to the point where even the creators have difficulty comprehending why their models make specific predictions. This lack of interpretability is a major concern, particularly in situations where individuals want to understand the reasoning behind a model’s output”

from - https://arxiv.org/pdf/2302.03494.pdf

"ASI" is still limited by physics and logic. Or if not, we're just more screwed, maybe. I suppose we're in agreement that in the long run (maybe longer than we think with some really cool stuff like Ems in between...), the likely scenario is we're screwed one way or another? I'm just glad people are working on trying to discover the chains of logic that let us not be so screwed, hard as that problem may be. I don't see any paradox there that makes the problem impossible, logic is logic independent of intelligence. Similarly, I'm very sure that no one else and no superintelligent AGI can ever reproduce the original string I used to produce the sha512 hash of a5136df53e73c4fdcde80d9ba41f2526c6e144e344c85a355f518b161f5b5171a6de831908ec32f6a6e503ae055219027d8e9890517977bbcd16a6cd9f36c858 (perhaps they'll find a collision). (Incidentally there are many secure systems and schemes that have not fallen. But you'd probably write them off as too narrow for some reason.)

Yes, there's "difficulty comprehending" current deep learning models, put modestly. There's also interesting work in interpretability/explainability going on to work on that. Nothing there says "impossible". However, even granting that it is actually impossible to explain and reason about current AI methods to the degree we need to in order to create chains of logic for alignment purposes, that just says that it's impossible to reason about an AGI produced using such methods, if it's even possible to produce an AGI that way. It says nothing about it being impossible to reason about an AGI produced using different methods.

> I'm just glad people are working on trying to discover the chains of logic that let us not be so screwed, hard as that problem may be. I don't see any paradox there that makes the problem impossible,

I understand you don't perceive the paradox. Nonetheless, my point is rather that when a paradox presents an impossible problem, then something about our premise is not understood. Or we need to find an entirely different concept to solve our problems. For example, an entirely different concept of intelligence not modelled strictly on how we perceive our brains. In summary, I think we are trying to solve the wrong problem. I might write something more depth at some point on that to explain my views.

> Incidentally there are many secure systems and schemes that have not fallen. But you'd probably write them off as too narrow for some reason.

This is precisely my point. We can build what is logically infallible to only be subverted by different means. If you can't hack the system, trick a human to give you access. This is how every such system has been defeated.