> it's that none of the math classes teach you how to think through and write proofs.

Formal logic is usually introduced in calculus and discrete math courses. Arguably though it could stand on its own especially if it was taught using modern computer proof assistants, which make the "structuring" of even fairly complex proofs very clear.