I don't get it. All these kinds of arguments make the assumption that humans are purely rational beings. They're not. Human beings are emotional, and if they're convinced emotionally that the AI should not be let out of the box, they won't let it out, even in the face of overwhelming rational arguments.

If you don't believe me, just consider how many religious people there are in the world (and many of them are very smart).

I'm guessing Yudkowsky only does the experiment with people he believes are rational... which shouldn't be difficult as most people interested in AI have some degree of rationality. Even agreeing to the protocol requires rationality:

Sure, someone could sit at the keyboard for two hours, repeatedly typing "I won't let you out", but Yudkowsky could warn the person that they are not actually "engaging" the AI for the allotted time period. If the person accepts this argument, they have some inherent rationality that Yudkowsky can exploit; if they don't accept it, he can argue that they didn't follow the protocol, so the experiment doesn't count.

I can engage in rational conversation whilst still letting my emotional decision stand by. In fact, it can even be seen as a rational choice - go into the conversation with the rational decision that you will not let the AI out of the box...

And people have done this, and they have won. If I am reading referenced article correctly, Eliezer had three wins, two losses before calling off the experiments.

He's not claiming he can convince anybody with his arguments, just that he has successfully convinced a few people. Make of that what you will.