You've probably met people you don't know are trans. If you saw Andreja Pejić or Laith Ashley in person there is no way you'd suspect them of people trans.

All you're saying is, you don't want non-passing people allowed in the wrong restrooms. Which, is... it's just a strange thing to want, really.