Physics does clearly "care" about the direction of time, because physics is a description of the physical universe. Dissipation, entropy, quantum measurement... etc. all have time as a key concept.

Math may not care (and it's true that most of the fundamental physics equations are often time reversible). But the universe (and any experiment to date) clearly shows that time has a direction and the equations work in that direction.