I found this:

the only way to know that a man thinks is to be that particular man. It is in fact the solipsist point of view. It may be the most logical view to hold but it makes communication of ideas difficult. A is liable to believe ‘A thinks but B does not’ whilst B believes ‘B thinks but A does not’. Instead of arguing continually over this point it is usual to have the polite convention that everyone thinks.

https://academic.oup.com/mind/article/LIX/236/433/986238

Yeah, that's exactly the passage. Completely unquotable, therefore I usually paraphrase a bit.