The more useful interpretation of the argument is that it's another form of justifying the social order through cultural perception of the inherent worth of people in each class
i.e. everything a wealthy person does thanks to wealth is traditionally promoted as a sign of their inherent worth, and everything a poor person does out of the conditions of poverty is interpreted as a sign of their fundamental roughness. This is especially true in a Protestant context of wealth being an indication of divine favor.
And rightly so. For two main reasons
- demonstrably negative expected value of the bets (like in casino floor games or the lottery)
- relatively high proportion of total net worth wagered
I'm very comfortable with labeling this as irresponsible (regardless of levels of wealth). It's not just a case of "everyone does it but only poor people are shamed for it", there's a clear distinction between the two cases.