There is no graph of speaker,conference edges that crosses an ocean in which this isn't an issue. Yes: it can be a bitch to get into the US to speak, even at serious academic conferences. I have friends who've been turned away at the border. But the same goes for US->EU, US->UK, US->JP, JP->EU, and even US->CA in one case I'm familiar with.

Major international border crossings just suck a lot.