The problem is that we don't have that transparency now, and aren't particularly likely to in the future. [1] Particularly for software-as-a-service: I can't audit Facebook's algorithms, despite their effects on what my friends and family see and discuss. And I can't easily audit the black box models used to make public policy, despite the value of transparency there, either.
[1] http://www.theatlantic.com/technology/archive/2015/01/the-ca...
Scale that up in complexity a few orders of magnitude, and the thing may as well be a human. That is, you'll likely have broad understanding of its "motives", but we have that for humans today and it doesn't help that much. I understand Facebook wants to sell advertising, but that transparency isn't sufficient for understanding why a particular message is delivered to your followers while another isn't.
Not if there are incentives for the algorithms to be subject to public scrutiny.
> And when the AI does reach human level (a long time from now), we may be no more capable of understanding its motivations
(1) You're implying they/it'll have "motivations" in the first place. My whole argument was about a lack of motives.
(2) There's a difference between understanding how a conclusion was reached and why it was reached. Example: You may not understand how your compiler decided that line X in your program is dead code, but you may still be capable of comprehending that it in fact is dead code.