It's less about the act itself (though they'd be called out for that too) but the immoral aspect is the fact that this is sex tourism.

You can ethically justify going to a Vegas or Amsterdam brothel under the pretense that the workers choose to be there and you're paying a fair wage for the service; it's a fair transaction and nobody is being exploited.

Shopping for prostitutes at a time and place where people are doing anything they can to survive in the wake of a disaster is exploitative.