In my grandmother's case, those trips to the casino actually greatly benefit her. At 94 years old, the casino provides a level of sociability and exercise that she doesn't get otherwise. That said, even the particular casino she visits most likely does more harm than good to the local community. Not by a lot, but by enough. Unfortunately, the perception of whether that crosses the line is going to vary wildly between individuals. Some will think it's not greedy enough, and some will think it's the equivalent of murder, and there's no legitimate way to decide where the exact appropriate middle ground is. That's something we all have to decide for ourselves, and it can't be forced upon others.
This statement is false. Governments and judicial systems around the world routinely decide where the exact appropriate middle ground is, and make binding judgements forced upon people by threat of fines and imprisonment, ultimately acts of violence.
Some of it is even democratic (deciding together, not each for ourselves).
In my opinion when there is an asymmetric power relation like that, then there is a question of responsibility. Especially if the incentive is to exploit the relation - which is almost undeniable in this case.
So I'm not making a statement about the general case and I don't think things can be boiled down to "this type of game is evil". It's the specific connection between these two extremes that I find worrying, or saddening really. But I'm 100% not the type of person who would deny your grandmother her fun. I think these types of problems start with things like education and attempting to heal the human connection between the powerful and the weak (sorry for the pathos).