You will never remember, or "passively recognize" any characters, (especially low-frequency ones) unless you specifically focus on remembering them. Also, at least 75% of words meanings become trivial once you learn what their corresponding characters mean. Or you can spend all your time getting all those similarly-sounding kanji compounds mixed up in your head because you can only write them in hiragana.