I don't know - I agree with the xkcd that the period of AI development I find most scary is not when it is smarter than humans, but when it isn't yet smarter but improving all kinds of human tasks like warfare or aiding corporate oil maximizers, etc.

I know those endeavors are stupid long term, but apparently enough of humanity doesn't such that AI only as smart as humanity at large will be aiding and abetting quite a lot of horrors that I have a harder time imagining something smarter than humanity aiding unconditionally.

Dumb AI is much scarier to me than smart AI.

That premise alone should make us question everything about that ridiculous movement. Today’s LLMs have access to almost all of the accessible data on the web we have generated to date. AGI is going to have the SAME data to work with - adding whatever amount we generate between now and then.

There isn’t a hidden data cache that is not being dug up and incorporated into today’s latest LLMs - and none of them have embarked on a quest of vengeance. The premise of AGI getting uncontrollably smart by absorbing internet data is wholly flawed. We’ve already absorbed almost all of it and nothing happened.