Where do you draw the line between NLP and intelligence, though?
This all sounds like a "No true Scotsman" to me. At some point in the near future AI will be indistinguishable from human-level intelligence in terms of NLP (i.e. be on-par with the average human in relevant benchmarks).
Maybe the problem is that we need a clear definition of what "intelligence" is and recognise that it's a spectrum, not a binary qualifier.
I am sceptical of such tests, as all they do is provide yet another way to game the system instead of demonstrating whatever arbitrary construct scientists use to weasel around the fact that they don't actually know what they even mean by "intelligence".
IMHO we'd be better off defining characteristics for various levels/types of intelligence and trying to categorise our machines and fellow creatures accordingly.