Confusion about this stuff is a sign that you didn't really understand it the first time. Believe me, I've been (am still) there...
Who needs groups? No one. Until one realizes that certain simple equation transformation rules are not based on natural numbers, or real numbers but on groups. Once something like that clicks in your brain, you can suddenly solve weird equations with sets and symmetric differences, or bitstrings and xor's or other pretty akward things which looked really scary earlier.
Who needs rings and semirings? No one, until one realized that a certain algorithm requires a structure... and this structure is a semiring! Thus, if one can prove that two operations and a set form a semiring, you can apply this algorithm without any effort, because it will just work! :)
Or, even more. Who needs the theory of katamorphisms, Anamorphisms and such? No one. Until one realizes how beautiful recursive datastructures are and how easy it is to program them once you understood the idea behind them (Check http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.41.1... for a nice paper about this)
Who needs integration (besides those numerical people)? No one, until one realizes that any simulation, most iterations and such are discrete intergrations in very, very akward and convoluted algebraic structures.
Ach, I somehow wish I studied math before studying computer science by now.
If it were taught as providing some insight or serving some purpose, it might be a lot easier to learn and retain.
Then when I got to college they introduced the notion of "program correctness" where you tried to prove, mathematically, that your computer programs were correct. This convinced me that computer science was simply a knock-off approximation of mathematics and I drifted away from it.
From your derivative example above I’d say it is this inexact nature of computer science that you especially like. Different strokes for different folks, I suppose.