Try changing how you take notes. I felt the same way until I discovered mind mapping.

Writing and linking concepts is almost like putting chapter markers in a video or flags on a page. It helps you recall what was said in what context.