The point of the Chinese room is to show you that the guy who has memorized the standard responses to various questions in Chinese does not in fact know Chinese. He's mindlessly parroting things.

what would it mean to "know" a language? one could imagine a series of increasingly complicated questions designed to relate various words, subjects, associations, maybe culture and history. But one could just as easily imagine a sheet of paper with the answers on answering them - and our friend answering them, in seemingly fluent Chinese. Im not convinced there is any experiment one could perform to convincingly separate the two (without removing the man or his translation aids from the box) - thus does your idea of "knowing" exist?

An AI cannot be removed from its box, because it doesn't have one. It really does have enough information inside of its essence to reply. In fact, that information makes up its essence.

I agree that in some sense their knowledge is distinct and of a different character to human knowledge. But what that means conceptually or morally exactly is very complicated, and cannot be dismissed easily