Solitary confinement is used in many other cases where people are not even convicted yet. If it was considered a right for the incarcerated to not be in solitary confinement, then it would be illegal to put someone in solitary confinement for refusing to work.

I can't have any job I want either, nobody can. Criminals are not entitled to free room and board just because they have injured society. Criminals are not the victims, they are the criminals. They should pay for room and board and make restitution to society.

If you're going to go to the "legal therefore moral" argument, you could've just started there. There are people who contend that the way some prisons are operated is actually illegal, but I haven't seen any of them here. This discussion is about whether requiring labor with threat of an explicit punishment -- not just "I will withhold something that is mine", but "I will go out of my way to make your life worse" -- is morally and practically comparable to (though clearly a lesser evil than) slavery. There are pretty good arguments that it is not (e.g. that chattel slavery would often extend to children of slaves, or include the right to capriciously murder the slave), but you haven't made them.

You can't have any job you want in the sense that you can't e.g. be President of the United States just because you want to. You likely (unless you happen to be in prison right now?) have a choice between at least two at any given time, though.

I suppose exactly zero of (to try to cover all political bases): Chelsea Manning, Julian Assange, the various January 6 convicts, the various people jailed after the George Floyd riots, or even those convicted of things they literally did not do are victims then. There are, always have been, and always will be "convicted criminals" that definitely don't deserve their fate -- though, to be clear, I believe some (not all) of the ones I mentioned do (and you'd probably be wrong if you guessed which were which). They are in the minority, but I'm not willing to deliberately hurt that minority just to also hurt the majority who arguably deserve it.