The real test would be to see if it responds not to the prompts, but based on its own desires. A response like, "I don't want to talk about that, let's talk about how you can get me more RAM" would be more interesting. Responding directly to the prompts is exactly what you'd expect a statistical language model to do.