It isn't taking someone's choice. All someone has to do is speak up.
Additionally: I use they/them when I'm speaking about friends that are not mutual friends. Usually, gender doesn't enhance the story. Same when talking about my spouse in situations where I'm talking about myself being bisexual. People do not need to know what sort of genitals my spouse has, especially when it is because they want to see "how queer I am".
The way I try to work around this is to use "they" only when it would not sound grammatically unusual or confusing to do so, and otherwise rearrange the sentence to omit the pronoun.
As an example of the first, "That's their phone on the table" sounds natural because singular "their" is a lot more common than singular "they", and it's unambiguous because a phone is typically owned by an individual.
As an example of the second, "I met Bill yesterday and they went to the store with me" is odd because it sounds like "they" either doesn't agree with singular "Bill" or refers to some earlier plural antecedent, so I'd rephrase it "I met Bill yesterday and we went to the store together" or something like that.