If not, how could you tell them from the "non understanding" robot?
I also think most robots would pass the "I couldn't tell its a robot" test with flying colors if they responded just like real people in the same settings as everyday conversations -- because those interactions are usually limited in scope, and you don't purposefully try to unmask those people with tricky questions.
Sure, my first question is always simple direct semantics test. Perhaps something like, what's the third word of this sentence if you start counting from the second word? Mitsuku chatbot that won the 2013 chatbot competition got this one right, but if I just slightly rephrased it to, "If I were to ask you what the third word of this sentence is if you started counting from the second word, what would you say?", it got it flat wrong despite having just answered it.
Reflective questions like this are good tests of understanding that are trivial for a person to answer, not so trivial for a system that doesn't truly understand what it's reading.
Honestly, I've always found any such construct absolutely trivial to identify. I've been able to stump every chatbot with my first question. No agent that doesn't actually understand would be able to pass a Turing test issued by a trained computer scientist, even if limited to 5 questions.