He's saying that working men are so looked down upon that the unemployed consider themselves a step above them. A bit like a perverse parody of aristocrats refusing to work.

This is not theoretical, I've seen it, I believe it. It is typical for the middle classes, especially the younger ones and what I can only really call kept women to look down on working class men as being one step removed from the prison population. Possibly feral. Never enlightened.

I think the primary cause is an education system that strongly gave the impression to young people that it was 'Yale or Jail'.

Basically the universities have created a class system by creating a new social class. Unemployment is less egregious than picking up a hammer or spade.

Otherwise you might be one of those ditchdiggers, you know, in "the world needs ditchdiggers too" quote.