The vision subsystem generates an embedding when it sees a cat, which the memory subsystem uses to query the database for the N nearest entries. They are all about cats. Then we feed all those database entries - summarized if necessary - along with the context of the conversation to the LLM.
Now your AI, too, gets a subconscious rush of impressions and memories when it sees a cat.
I struggle with the Chinese Room argument in general because he's effectively comparing a person in a room following instructions (not the room as a whole or the instructions filed in the room, but the person executing the instructions) to the human brain. But this seems like a crappy analogy because the better comparison would be that the person in the room is the electricity that connects neurons (instructions filed in cabinets). Clearly electricity also has no understanding of the things it facilitates. The processor AI runs on also has no understanding of its calculations. The intelligence is the structure by which these calculations are made, which could theoretically could be modeled on paper across trillions of file cabinets.
As a fun paper napkin exercise, if it took a human 1 second to execute the instructions of the equivalent of a neuron firing, a 5 second process of hearing, processing, and responding to a short sentence would take 135,000 years.
I'm only today learning about intentionality, but the premise here seems to be that our current AI systems see a cat with their camera eyeballs and don't have the human-level experience of mentally opening a wikipedia article in our brain titled "Cat" that includes a split-second consideration of all our memories, thoughts, and reactions to the concept of a cat.
Even if our current AI models don't do this on a human level, I think we see it at some level in some AIs just because of the nature of a neural net. Maybe a neural net would have to be forced/rewarded to do this at a human level if it didn't happen naturally through training, but I think it's plenty possible and even likely that this would happen in our lifetimes.
Anyway, this also leads to the question of whether it matters for an intelligence to be intentional (think of things as a concept) if it can accomplish what it/we want without it.