I got all the rules (declensions, conjugations, etc) but I always had trouble piecing them collectively. A lot of the declension endings would be the same (such as 4th declension nouns) and it was hard for me to differentiate between a nominative and accusative in the context of a sentence. Hence I never studied Latin in college. I'm sure the world is a much better place for that. Wow, I can't believe I am reopening old wounds from 12 years ago ;)
I genuinely believe the single perfect test score is also empirical proof of the difficulty of Latin in an English-speaking world...
I would much rather parse MIPS assembly all day than read Latin.
One of the more interesting hypotheses suggested was that Latin classes, unlike a lot of SL classes, are not nearly as "immersive" as more common ones, and a lot of the language processing hardware we've got is better at digesting from spoken immersion. (Part of that, of course, is that outside of specialized environments, we don't really have any "native" speakers; part of that is also that common "spoken" Latin was a lot simpler than the brick-by-brick construction of Vergil or Cicero.)
I also agree that the Vergil and Cicero constructions are harder than what you would hear at a spoken Latin gathering (or probably in a Latin-immersion class). The word order people usually use when speaking is more like modern Romance languages (with some tendency to put the verb at the end, but not, say, chiasmus!). That did make me think that ancient Romans probably didn't talk like Cicero or Vergil either. :-)
I always remember this line from Cicero
Magna dis immortalibus habenda est atque huic ipsi Iovi Statori, antiquissimo custodi huius urbis, gratia ...
in which "magna" 'great' modifies "gratia" 'thanks', which is the subject of "est habenda" 'should be given'. That's tricky, or at least a lot to keep in your head when trying to parse it.
Did you study Latin for several years before? I think that class is typically intended for people who have already studied Latin at the high school level for three years (most likely reading Caesar or other authors first). That would give a lot more opportunity to become familiar with Latin grammar rules.