But the only case where you would rely on group membership is if the individual traits are unknown. Obviously generalization will never be as accurate as individual knowledge. Why would anybody be worried that people would act otherwise?

Assuming things about individuals based on group membership is a key part of racism, sexism, etc. Why would anybody be worried? Because people do it all the time.

Stereotypes exist because there are statistical patterns in groups of people. Those stereotypes are useful cognitive tools while they remain accurate. It certainly isn't fair to people in those groups who don't fit the stereotype, but as long as the stereotype accurately describes the majority, there's nothing to be done about it.

Would you run up to a lion or a tiger in the wild and try to hug and kiss it? I'm guessing you have a stereotype of big cats as dangerous predators, and you'd steer well clear, even though there are countless friendly lions and tigers in captivity that have never hurt anyone.