Yeah and at some point the remaining people who haven't formed any kind of natural immunity many just be "hard to infect". Their ACE2 receptors might be different, or their innate+intrinsic defenses shred the virus very effectively before infection really gets going. They just aren't accommodating hosts for the virus due to some trick of their own biology/genetics. Selection effects work both ways.