1) your direct language ("a man who identifies as") is problematic in the way it frames the argument. It is both dismissive in that it implies the rights being stepped on are trivial and not worth concern, and in that it up-front implies (again, you lead specifically with "men who...") that the people involved are claiming these rights in bad faith.
2) Your dismissal of the defense of rights for a marginalized group is problematic. I explained why. I don't think I should need to continue to rephrase that argument.
Calling your comment transphobic should probably lead to you in good faith wondering why people think you are using transphobic language; if you want to not be perceived as transphobic (and if you think you are okay with trans people) then figure out why your behaviors do not line up with that belief about yourself and fix it.
If it is merely a question of priority ("there's more important things than the rights of the marginalized to focus on right now!") then: if not now, then when? You might have a good answer in this instant, but... the goalposts always move. Since marginalized groups are by their very definition small and at the edges of society, there will always be something to prioritize over their rights; that is how marginalization perpetuates. If we do not defend the marginalized and most vulnerable intersections of our society, then our society has failed.
"A man who identifies as a woman" is a disingenuous statement--it is leading with some sort of made up notion of what their gender should be ("a man who..."), which is a false framing in the first place. Transgender women are women. They are not men with a qualifier.
Moving on: transgender bathroom rights matter. Access to public bathrooms is necessary for access to public spaces and is furthermore generally an indicator of free unencumbered movement in public. Therefore, state-sponsored humiliation tactics are indicating a stance that trans people are not welcome in a public space. This is problematic, and furthermore is certainly a matter of civil liberty. The ACLU should take those cases and it's important to stand up against steps to marginalize and "other" groups of people. As public space is necessary to exercise certain rights in the constitution, particularly the first amendment, this matters at a very deep level.
To give a clear and specific example: a friend of mine is no longer safe visiting his family in North Carolina, as he is trans. He must either choose to violate the law in public to fit in (he passes as a man), or by following the law out himself or put himself at risk. He no longer has the liberty to visit his family like you or I do, without additional--potentially severe--legal or physical risk.