While this is true, at least in the early pandemic when social circles were pretty tight (often just your household, but even if not quite a bit smaller than during normal times), the virus has to be introduced to that social circle, it won't just manifest itself out of nothing in one of it's members. Masking when not interacting with that circle will help keep the virus out of that circle of people in the first place.
This is a very hard thing to maintain for long periods of time, but in the context of 2020 it's a reasonable approach to take.
as i see it, masks make sense in 2 general cases:
1) when uncertainty about transmission and lethality is high, as was the case in nov/dec 2019, until population risk is more closely bounded
2) when situational risk is measurably elevated over baseline, like having comorbidities, surgery, packed mass transit, etc.
by april 2020, we (in the US at least) had enough information that masking wasn't going to be that panacea, when taking human behavior into account (trust channels, weariness, etc.). yet, it was still weaponized by the media and politicians.
i know this because i was making these very same points back in april 2020. to be effective, a mask has to be worn at the right times (like being face-to-face with another human) and correctly (creating a tight seal around the nose and mouth using an effective filtering material). most people weren't doing either of these most of the time.
the CDC was initially correct in assessing that (the way we wear) masks would not be a panacea for covid, but that was politically unpalatable which is why it was walked backed so haphazardly. masks, and all the CDC outrage around them, was effectively mediopolitical redirection of our anger toward each other rather than toward our corporatocratic institutions to quickly and correctly assess the threat and then appropriately deploy (only) effective measures.