I've experienced similar, no matter the class start time. I think I'd agree with your assessment of "you're in the wrong place" not necessarily because of the subject matter, but because of the presentation and environment in which it occurs.

For example, in my high school AP Calc class, I often couldn't stay awake because I picked up on the concepts very quickly whereas my peers did not and I would quickly get very bored when being lectured on the same topics day after day. Contrast that with the amount of books on Mathematics I currently own, that my favorite language is Haskell and that I currently am employed writing software.

It was almost as if my body had developed a heuristic for how useful today's class was going to be and reacted accordingly. The same sort of thing happened through college (Psych, Business, etc) whereas I was known to read psychology textbooks for fun when I was very young.