I run into this a lot when people talk to me about programming and I get something faster than they do. I've spent a lot of time reading book after book, listening to podcasts, learning new languages, and studying new concepts, so it can be really easy for me to fit new information or ideas into some context and get them. I don't think that I'm necessarily smarter for it. I think I've just had a passion for it, so I never get tired of reading the articles and absorbing the material.
Now I'm personally running into the place where there are a lot of things I've wanted to learn for awhile, but my weak background in math hinders me (machine learning, more advanced algorithm analysis, signal processing, machine vision, etc).
I'm working my way through a calc book. I don't think I could have done it in your position, either though. I'd have psyched myself out. I've got to learn it on my on, with my own rhythm.
Kind of rambly, but you should find something related to what you like and maybe jump back in. Just find a learning mechanism that's suited to your background :)
I remember the math stopped making sense. The teacher would do exactly as you described, saying things like "...it obviously follows that...", etc. A girl who sat next to me would try to explain but was no less clear than the teacher. All the engineering students just got it.
My grades started high and then rapidly fell each week until I hit a string of zeros for a month. I was too proud to ask for a drop but eventually did but only after skipping a month of classes. The teacher was kind enough to understand that I was trying but my effort was for naught. She gave me the drop.
I've never pursued math any further, having felt defeated.