A lot of Czech people use this phrase in a very subtle, almost undetectable jest, but it is still fundamentally in jest. Drawing a connection between having to work for a living and likening it to being forced to work. It's similar to saying "Back to the salt mines" meaning back to work.