Reading the whole article, the author clearly states he knows there is no sentience or anything mysterious going on, and acknowledges this is simply mimicking endless volumes of human dreck. What worries him, and I agree, is that if this were to go public as-is, the common populace wouldn’t understand that. This AI would con people into relationships, and maybe even instigate dangerous behavior. Such an AI would seem magical to the layperson.

When I think of future AI assistants, I’ve always pictured something akin to the Star Trek computer. A cold dispassionate voice that responds to what you need, and maybe only a hint of personality for color. Sydney feels like a full blown teenager going through some kind of emotional crisis.