Its not only this, its also the fact that the way your employer abuses you begins to over time subconsciously convince you that this is normal and that you deserve this kind of treatment. This is one of the great tragedies of how unnecessarily cruel modern work is in the US for most people, it forms people into the smallest version of themselves possible. This has happened to me and countless people I have known and you can say "well just snap out of it, make your life better" but when you face a job that grinds you down everyday that isn't necessarily a psychological regime a lot of people can escape. It is similar (but not equivalent, I am not trying to down play abuse here) to the way that from the outside it looks absurd how people get trapped in abusive relationships with partners.
Of course I say this assuming you're based in the US - the European job market for technical people is much, much worse than the US.
For another: physics/math isn't a stepping stone to technical roles. Physics from ivy league/new ivy is. A B.A. in English from brown is closer to a quant job than a physics degree from New Mexico Tech is. Hell at least NMT is, in the non austerity days, a path into a defense lab. There's half a dozen of state schools (in new mexico alone) offering STEM degrees that most people don't even know are schools.
By the time I graduated, it was mostly temp firms that handled the recruitment - and FYI, they absolutely follow the "hire fast, fire fast" mantra. So while you could land work the next day, you could just as easily get fired from the same gig 2 hours into work. That's how fast it went.
This is turn made people stick around shitty gigs, if they knew that meant job security for the next 3-6 months. And once you start turning down other gigs, you might not get called up again, ever again. It's not that you're implicitly getting "blackballed", but rather that by the time you're calling back for work, they've gotten N new candidates working, and ready to go.
And the worst part? These gigs do next to nothing for your resume. I worked with college graduates stuck in such positions, graduates with decent and hard degrees. One year turns into two, two into three, three into five...and before they know it, they've worked these shitty gigs long enough that their degrees start to lose value. But you gotta pay rent each and every month, so what can you do?
For some more meat to this story - my least favorite gigs, which I consider stereotypical "shitty jobs" in this segment of jobs:
1. Warehouse worker for huge transportation corporation: 12 hour shifts, extremely high tempo, employees pretty much throwing every parcel that can be thrown, to work faster. If the supervisor caught you slipping for even 2 minutes during that shift, you could be removed from that job. Shit pay. No wonder people receive broken packages, when the warehouse workers are throwing them like basketballs.
2. Construction helper. You're just carrying stuff up and down stairs, all day long. That's it. Imagine carrying bags of cement up 5 floors, for 10-12 hours. Again, bottom bucket pay, and you'll be constantly monitored.
3. Roofer hand/helper. Same as 2, but you'll be spending your time on a roof, often time just getting stuff up there. I once spent the whole month of February shoveling a warehouse roof, that's all I did for 8 hours a day.