They're also in another country that's highly male power oriented, probably talking to a male head of a household, potentially to someone wealthy and even powerful. Meanwhile you may be significantly smaller, may be female (this isn't meant as an insult... just that you may have less muscular mass/strength), and are in a country where someone may claim you stole your passport from their possession and they may literally cut your hands off, where the authorities may even agree with this should you go to them. Or, at the very least, they may no longer sponsor you and you may go home to your starving family with few other options. Some may argue that means the option you had was better, I think it's an argument to promote modern slavery and exploitation.
My situation and I suspect yours are not remotely the same to these folks. If someone snatched my passport, I'm a fairly good sized and a reasonably strong male to muscle myself in (I also have a concealed carry permit, so I have a weapon should I need it). I have several more explicit rights and resources at my disposal, I have a government that issued my passport that will look at it say it's clearly my passport. My stance in such a situation is significantly different, assuming it happened here in the US.
Meanwhile you have examples like Britney Griner who had similar advantages to me, arguably even more given her fame, and even she was detained and at the mercy of the Russian government where we literally had folks like Tony Blinken bartering a hostage exchange for her return. I'm not sure that would be the case if I was detained in Russia on similar charges (then again I probably wouldn't have been targeted as readily, either). The power dynamics in these situations aren't so dry cut but I assure you, these maids have very little power and rights. I recently had a layover in Qatar and even I was a bit weary of that--I tend to like to have layovers in more progressive countries where I'm less likely to deal with some abusive government.
Griner though? She broke the law and got arrested. And she's not a desperate person. I don't see the similarity.
There is a "breaking" period for most of these workers where they are demoralized as well.
Before that, employers had the right to do so and it was (might still be) the norm.
Refusing to hand over your passport is like refusing to hand over your passport at immigration.
So if a migrant worker called the police and said "my passport has been stolen" they'd be laughed at (and possibly punished for wasting their time).
I once worked at a company where some had to go to Dubai to install one of our systems. Some of my more knowledgeable colleagues refused to go, and after hearing their stories, I would also never go.
I guess that's my answer though. Things are just drastically different in some places and the authorities are on side to support the perpetrators rather than the victims.
I'll leave this as a small taste of what can happen: https://www.bbc.com/news/technology-31692914.
Who says you're the victim? In their eyes you're the perpetrator.
I was a bit confused, but as a junior developer I wasn't going to make a scene for that,.when apparently nobody from the group was blinking at the request (perhaps everybody was thinking the same?).
Luckily a senior manager from $bigcorp barged in and when asked to hand his password just confidently yelled "there is no fucking way I'm going to give you my passport" and walked through. We followed.
How does this work? As an adult, if someone tries to take my passport I would refuse. If they snatched it out of my hands, I would take it back. If they somehow got the better of me and hid it or locked it up, I'd go to the police.
I could see maybe an abusive situation where going to the police isn't an option because the victim is locked in the house. In that case I'd expect that when the perpetrators are caught they'd be charged with kidnapping for that. And then some other charge (theft?) for locking up the passport.