It totally is about the mindset. And it's a combination of being properly motivated(intrinsic interest), and also seeing different ways in which to navigate the subject, and also knowing which ones are suited to your abilities at that moment.
With all of those traits, it's easy. If you miss one, you might be able to slog through. And if you miss two, it will be extremely difficult.
And our educational systems really don't encourage anything but memorization, despite ample PR otherwise, so there's a selection bias by the time you hit college level courses.
I've had similar issues as the OP, though at university I accepted lower marks than I could have gotten because I simply didn't want to go to class. I could learn the material on my own and use the time-savings to learn other things I was interested in. I almost always made top marks on exams, so it wasn't an issue (unless attendance was mandatory...)
I met a few people that had screwed off in High School, but for some reason had decided to take College extraordinarily seriously. I realized that for other people, material that would take me an hour or two to learn would take a solid 8 locked in the library.
As it turns out, after talking to some of these people, the problem wasn't that they weren't smart. It's that they didn't know how to learn, or had some misconceptions about what learning entailed. I've watched people slog through upper-level math classes by trying to memorize the relevant material, when they could have understood it in half the time, if they'd approached the material with the right mindset.
For a while, I was the OP. At least until I realized that if your goal is to beat your peers, or measure up to some arbitrary external standard, you probably won't. Even if you succeed, you'll make yourself miserable trying. On the other hand, if your goal is to learn, you'll do it. You might even stand a chance of beating out your peers. Unfortunately, something a lot of people don't understand about learning is that after a certain amount of effort, it just takes the amount of time it's gonna take. You can't really sit down and say "I'm going to learn x in y hours" and not be disappointed much of the time. Sadly, the education system all the way through College makes it seem like this is so.