I genuinely think you don't need a math background to get a general understanding of category theory, because it's so high level. A lot of it is just drawings, even, not complicated formulae and proofs. It's not category theory itself that's difficult to understand, it's the examples and vocabulary.

I think you really need no more than high school algebra to understand basically what category theory is about.