Originally, sure, but I think it's been perverted over time (in the USA at least) to the point that it's really only applied to the destitute. Also, I think it's become enough of a cultural norm that the non-religious (or those subscribing to a different religion) still feel some pull towards the concept since they were raised where it was prevalent.

It also comes down to it being easier to spot someone in poverty than to spot someone that's rich but hasn't worked for it themselves. That might explain most of it.