You overlook the inherently collective nature of labor regulation. Labor regulation seeks to prevent individual choices from creating market-wide pressures and a societal race to the bottom. It doesn't matter if being asked to pitch in on weekends without extra pay "doesn't bother you." We ban you from even voluntarily agreeing to do that because we don't want to be pressured to do that too. Individual choices are restricted to create a floor for everyone.

From that perspective, any difference between "type of work" and "working conditions" is wholly artificial. People don't want to be pressured to work on weekends, they don't want to be pressured to do sex work, etc., so we ban those things to eliminate those pressures.