Another math major here (in my final year). My professors routinely use square brackets after the second or third level of bracket nesting.

I think it must depend on the professor/university.