Yes, Chinese people who went to school do mostly know the characters. And yes, given that knowledge characters are more helpful to the reader than a phonetic writing system is.

But memorizing a decent inventory of characters is a process of several years, and the boost in ease of understanding is under normal circumstances too small to notice. I notice that communicating in characters is easier because I'm not a native speaker of Chinese, and the semantic content the characters come with helps gloss over my lack of vocabulary and poor grammar. I can already understand English; I would get minimal help from extra semantic annotations to English text.

The total savings in terms of ease-of-reading in Chinese characters compared to an alphabet, summed over a person's entire life, will never pay back the effort it requires to learn them.