But the Turing test (which I concede, LLMs do pass) doesn't test if some system is thinking; it tests if the system can convince an unbiased observer that it is thinking. I cannot come up with a better "is this thing thinking" test, but that doesn't mean that such a test can't exist; I'm sure there are much smarter people then me trying to solve this problem.