Maybe emergent algorithms like biological evolution never “understand” the thing they have made, but the thing still has been made. The same might apply to our steps toward AGI.

We can’t even predict all the emergent patterns in current AI much less future, more complex systems.

Maybe there never has to be a time where someone sits down to create some sort of “feelings module” for something akin to feelings to emerge in a complex system. Same, presumably, even for sentience.