Nutritional deficiencies. Aging. Lack of sleep. A variety of diseases. Use of certain drugs (like marijuana).
All of these have been documented to affect memory.
Also, I think one's attitudes towards the past will affect one's memory of it. For example, I am very present- and future-oriented. I rarely dwell on the past or review my day (or my past life in general), I don't keep a journal. I also don't ascribe much importance to most day to day events. As a result (combined with some of the other factors I listed above), I don't really have a great long-term memory. Many of the things in my childhood, and even up to college and beyond are lost to me. Even when I've kept a journal, when I've gone back, I didn't remember many of the things in it. However, I think if I reviewed my journal from time to time, thought about what happened in the past more, or made an effort to remember it, I'd have a better long-term memry. Spaced repetition.
'Having loads of memories' is not exactly the same thing as 'learning more' or 'having clear focus on a goal', or 'intuition', or 'improved decision making', or 'being able to do more things', or 'knowledge', or 'wisdom' - and "I wish I remembered everything" sounds more on the lines of anxiety or hoarding, than a genuinely useful thing to desire.
And that wasn't said in the context of learning a subject, it was in the context of "remembering moment by moment thoughts" and "living well". Specifically, how do you live more-well by remembering what you thought about music one day four years ago when watering some house plants?
The answer is, you don't know, because you can't remember any of them.
You remember everything about your life - for what benefit/purpose/reason? Even if we could remember everything we don't have the time, attention, or brainpower to dutifully filter and process three hundred million seconds (~10-20 years) of memories before making a decision, every decision.
You write down all your interesting thoughts and ideas - for who, for what? Other people have already a lifetime of thoughts themselves and more they wish they could think.
Is it almost more interesting to forget, and then you can re-experience things anew and not get bored of 'seen that, done that, been there'?
I wonder if one meaningful way forward is artificial brains/compute engines (probably required to be sentient AI - but maybe not?) which can combine and work on the thoughts and scale of thousands, then millions, of people.