I held off taking physics in college until my senior year because I dreaded the 3 hour labs. By the time I got there I had 3 levels of calculus (and linear algebra, and plenty of proof based CS classes) under my belt, and it turned out to be one of the easiest classes I've taken.

Like you said, I didn't have to bother memorizing anything, because everything could pretty much be done with simple integration or differentiation.