So I simplified the problem, by replacing the ability to understand Chinese with the ability to see in colour: http://www.fmjlang.co.uk/blog/ChineseRoom.html
While the simplified system is able to distinguish colours, how does it have the first person experience of seeing in colour?
So yes, the guy in your room can experience colors by using the sheets (though in a different and much less capable way than most of us, thus him not wanting to equate this with our common understanding of experiencing color), and is color blind without. And I kind of get deaf when plugging my ears, though that experience is vastly different from someone who never had hearing to begin with.
That doesn't answer what it is like "seeing red" and "feeling blue". I don't know what it's like for you. I can only look at your behavior, see it being similar to mine under the same circumstances and thus assume your internal state to be similar. Until your behavior tells me it being different... like someone blind running into a wall or someone approaching a chasm with no signs of fear.
ps: had to google "feel blue", as it meaning "depressed" isn't really a thing in my native language. Translating (color based) idioms is actually kind of telling... it seems only possible directly if both cultures had shared aspects / experiences.
So yeah, while conflating consciousness with intelligence doesn't make a lot of sense to me. But I'm not sure equating it with perception is getting us anywhere better (hope idiom didn't get lost in translation).
The person in my room (an achromat) can't experience colours at all, and they're able to tell you that. Their eyes have no functioning cones. They see everything in monochrome, so to them, watching a colour movie is how watching a black and white one is to us. Under low light conditions our cones are inactive and we see only in monochrome using our rods, just like achromats do. So people with normal colour vision can experience what it's like to be achromatic, but achromats can't even imagine what colours are like.
In the room, the person can only tell what colours the papers are by observing how dark they look when overlaid with the coloured filters (whose colours they know).
The metaphorical use of colours in the language isn't really important, and as you point out is dependent on culture.
There is a sense in which the system has an experience (necessarily first-'person'), but it is nothing like what we experience on seeing color. It is no more or less an experience than an ordinary color meter would have, and it lacks any awareness or concept of having an experience, no memory of it, no recognition of a similarity to previous experiences or of it being unlike any previous experience, no ability to imagine such experiences, no expectation of future experiences, and no emotional reaction to it ('my favorite color!') It is an experience only in the weakest sense possible.
I prefer a simpler analogy: what step in a sorting algorithm actually does the sorting? The question is actually nonsense, because every step is necessary, and removing any single step will break the sorting property for some inputs.
When you have all the relevant information, it's clear that the original question itself is ill-posed. We lack some of that information when it comes to.consciousness, hence why the Chinese Room is convincing to some.
Consider performing the algorithm of Alpha Go Zero through a million people equipped with pen and paper. One can hardly argue against this being possible in principle, but what is playing the game?
This is not proof that AGI will automatically be conscious, or even just that some sort of conscious machine intelligence is possible, it is merely intended to suggest that Searle's line of argument does not show its impossibility.