Could you elaborate on this? Mathematics, perhaps uniquely among human intellectual endeavours, has no hidden rules. Its rules may be complicated and unintuitive at times, but they can be listed; and whatever follows from these listed rules is true (to be precise: whatever follows from the axioms is true in any model of the axioms), and whatever contradicts them is false, regardless of how natural or un- it may seem.
In particular, I am curious what you mean by "There would always be rules that were often forgotten." Do you mean that you forgot them, or that your teachers forgot them (both of which were eminently believeable!), or that practicing, professional mathematicians forget them? I am sceptical of the last (although plenty of, indeed probably all, professional mathematicians don't need to keep all the rules explicitly in their heads at all times), and, although I am eminently willing to believe the first two, I think that that shows a fault in the teacher, not the subject.
It seemed that mathematics required more of my time than any of the subjects. So wouldn't spend the necessary time needed to actually store it in my head. Later in life, I realized how fascinating math was, and fell in love with it. Even though I am still not good at it, it's fun to solve puzzles.