When I was a student in University we used Formal language and automata theory to proof that algorithms are correct. But we studied computer science, not math.