I say charge more, take higher deposits if needed but be more hands on with all speakers (not discriminating against "global south"): get their precise arrival info, always be in touch about their status, meet at the airport, organize their accommodation, etc. So they have no chance to escape your hospitality
Life does happen in any case--especially in these days when people are acutely aware that they can't show up at conferences with an illness. I've also filled in for a speaker who just totally forgot they had a session at the conference and were double-booked elsewhere. It happens. Might seem odd but it can happen when you might have a a dozen speaking slots a year and many more proposals in flight.
In fact what I was replying to was someone saying it was an event organizer's "duty" to write invitation letters for conference attendees for Visa purposes, which it is objectively not. The country is irrelevant, the economic development of the country is irrelevant.
I very much agree it's exactly organizer's duty to write visa invitations, and without prejudice and discrimination, and if they want they can do other things to combat abuse as I suggest.
Whereas your idea that "it's not his(sic) job to do anything" is very dubious on many levels and no offense I hope I don't have a misfortune to attend a conf you organize :)