"My reasoning is not the cognitive overhead that's caused by memorizing tons of characters, but overhead of encoding/decoding the affixes into that type of writing system."

But that need to memorize IS a cognitive overhead that is hard to dismiss. The Japanese have to learn thousands of Kanji from their early years (of elementary school) till well into high-school! And I would say that the encoding/decoding is actually an overhead only until it gets automatized in subconscious mind.