If a racist hermit never interacts directly with anyone, sure that is harmless. But if she actually just racially isolates herself (which is pretty hard to do in many parts of America), then she still has influence over the institutions around her. If someone then wants to join her racist gardening club, then her racism ends up affecting things.