>If we had a USB or negative income tax system then that benefit would be clearly spread across the whole population evenly

But we do? The EITC is basically a negative income tax.

https://en.wikipedia.org/wiki/Earned_income_tax_credit

>I think the argument is that their labour is being subsidised which amounts to a benefit for BigSam.

Ironically negative income taxes are closer to a labor "subsidy", than food stamps, because they're contingent on income (ie. labor), unlike food stamps which don't give you more money if you make more.