There are many definitions of sex. Some would say that sex should be defined chromosomally. I think it would be very unhelpful to put an M on a passport for an XY cis woman -- it should be an F, or maybe an X, if X were just a marker to indicate an unusual edge case. If so, shouldn't transgender people at least be afforded an X on their passport?
And then there's what's between your legs. Well, this is surgically alterable to some extent, so do you mean what's currently between your legs, what appears to be your legs, what was between your legs at birth, etc.? Or do you mean reproductive function? Maybe someone might say that a regular surgery isn't sufficient to change sex, but stem-cell-grown genitals do count as changing sex.
Regarding outward appearance (etc.) being correlated with "sex" (however it's been defined) -- that's exactly my point! In fact, because outward appearance is what is important nearly always -- except possibly in dating, reproduction, and sports -- and because there are a plethora of different meanings when sex is defined as "biological truth," I honestly think that outward appearance (etc.) is what most people actually mean when they say someone is a "man" or a "woman." Therefore, sex, in common usage, is appearance (etc.). This matches with how a tomato is a vegetable culinarily (i.e. colloquially), but a fruit botanically (i.e. "biological truth").
(Anyway, all this is really beside the point of this thread. You can disagree with everything I've said -- I'm just trying to argue that the U.S. is presently a difficult destination for transgender travellers.)
There are many other situations where sex and its material consequences matter more than gender. Yes, it’s not gametes that make a man run faster, but there’s a clear causal chain between gametes, sexual differentiation pathways, gonads, and testosterone.
Incidentally, it’s a really weird idea that we can’t tell someone’s sex without looking in their pants, or without some advanced scientific analysis. In reality, outward appearance is extremely highly correlated with sex, and humans (and a lot of animals!) have evolved to be experts at detecting sex, sometimes at a great distance. Visual appearance, gait, voice, behavior, and more, are enough clues to not just make a guess at sex, but to do so with very high accuracy. Probably upwards of 99%. Official documentation is there to clarify the 1%.