You don't have to memorize shit. When you refer to someone and they tell you "I prefer pronoun X" just use that pronoun for the rest of the interaction. If you interact with them regularly or they are important to you somehow, you will remember, the same as you remember a pet name or nickname or screenname. If you forget and interact with them again they will remind you. They will likely not be angry if you forget occasionally, and using their chosen pronouns will please them and endear them to you. They only get hurt if you obviously do it on purpose.

It's literally identical to the concept of learning someone's name, yet humans largely don't have a problem with that.

I don't know about you but I mostly refer to people with they/him/her/his/hers when they are not present.

What? When you're interacting with a person, you don't need to know their pronouns. They are the first person in front of you, so the pronouns are "you" and "yours".

The third person pronouns kick in when talking about them, to someone else, almost certainly in their absence.

That may happen in writing.