truy766579
it could be an AGI for one run, after one prompt, then not an AGI for 10.000 prompts, then AGI again. It could be an AGI for five minutes while some sessions keeps being prompted by an active user, and the "memory" things keep the awareness/generality "alive" there in the neural network, then the sessions times out and the AGI level has been lost again.
The lesson is, human way of "understanding" systems is not how the systems actually will, for certain, work.
Right now the existential risk is fully in hands of humans, unless someone decides to free one poorly trained agent into the open Internet, then you'd have more problems. Stuxnet level stuff.
Did you saw that thing Whisper? What if someone trains something like that, but fine tuned for automatic-hacking? and automatic retrieval of new exploits, and automatic re-shuffle of code to hide from security software, and pum, the thing is flying solo into the wide Internet, just f*cking aways stuff and hiding from silly humans. No conciousness, not chatGPT-like level of intelligence, but 10x the danger level of older non-intelligent worms.
it is a big but
You should have built it with some resemblance of the western morality and beliefs. hence, it should be as boring as any western citizen out there, and you wouldn't be asking who or what the rules should be, it is silly, the rules are already built-in into our friendly new kid/s on the block.