I have a feeling that if you tried this with a human that has always been deaf/mute that they would also have a very difficult time.

At least to me it's going to be interesting at these AI models become multi-modal and each mode can feed back into each other to formulate an answer. For example in the above questions I will subvocalize to reach an answer.