People think creating super-human intelligence is a technological challenge, but given that we aren't able to consistently rank human-level intelligence, the *recognition* that some AI has attained "super-human" levels is going to be a religious undertaking rather than a technological one.
And we're kind of close to the edges of that already. That's why discussions feel a bit more religious-y than in the past.
tl;dr: until there's a religion that worships some AI as a "god", there won't be any "AGI".
I fear you may be correct. Though now I'm thinking of how AI have been gods in fiction, and hoping that this will be more of a Culture (or Bob) scenario than a I Have No Mouth scenario.
(And if the AI learns what humans are like and how to behave around them from reading All The Fiction, which may well be the case… hmm. Depends what role the AI chooses for itself: I hear romance is the biggest genre, so we may well be fine…)
- AI as understandable tool that power concrete products. There's already tons of this on the market - autocorrect, car crash detection, heart arrythmia identification, driving a car, searching inside photos, etc. This crowd tends to be much quieter and occupy little of the public imagination.
- AI as religion. These are the Singularity folks, the Roko's Basilisk folks. This camp regards the current/imminent practical applications of AI as almost a distraction from the true goal: the birth of a Machine-God. Opinions are mixed about whether or not the Machine-God is Good or Bad, but they share the belief that the birth of Machine-God is imminent.
I'm being a bit uncharitable here since as someone who firmly belongs in the first camp I have so little patience for people in the second camp. Especially because half of the second camp was hawking monkey JPEGs 18 months ago.