Just imagine a world where you'd never have to hear "because I said so" and "becasue fuck you, that's why" ever again. A world where for everything that goes wrong, there'll always be a clear way to fix it and some clearly identifiable someone liable for the mistake. A world where being handsome is no more an advantage than having blue as your favorite color. Etc.
Maybe it's not perfect, but it would most certainly take away power from people that don't deserve it. I can see why the "powers that be" would try to convince us that this is "dystopian".
And when the AI does reach human level (a long time from now), we may be no more capable of understanding its motivations than we are of understanding those of other humans today.
Not if there are incentives for the algorithms to be subject to public scrutiny.
> And when the AI does reach human level (a long time from now), we may be no more capable of understanding its motivations
(1) You're implying they/it'll have "motivations" in the first place. My whole argument was about a lack of motives.
(2) There's a difference between understanding how a conclusion was reached and why it was reached. Example: You may not understand how your compiler decided that line X in your program is dead code, but you may still be capable of comprehending that it in fact is dead code.
The problem is that we don't have that transparency now, and aren't particularly likely to in the future. [1] Particularly for software-as-a-service: I can't audit Facebook's algorithms, despite their effects on what my friends and family see and discuss. And I can't easily audit the black box models used to make public policy, despite the value of transparency there, either.
[1] http://www.theatlantic.com/technology/archive/2015/01/the-ca...
Scale that up in complexity a few orders of magnitude, and the thing may as well be a human. That is, you'll likely have broad understanding of its "motives", but we have that for humans today and it doesn't help that much. I understand Facebook wants to sell advertising, but that transparency isn't sufficient for understanding why a particular message is delivered to your followers while another isn't.
Instead you'll hear "sorry, but it's the algorithm. I understand you need your paycheck, but there's nothing I can do and the programmer won't be available until the middle the of next week".
I learn so much from fiction.
I do understand why people don't like it. Everyone feels that an algorithm could screw them over in a situation where a person wouldn't. It's the driverless car argument. And we all know the response, that humans aren't perfect either and we have better metrics and tools for understanding and fixing machine mistakes than we do human mistakes. The same is true for algorithms that control Tinder, credit applications, health insurance rates, et al. They aren't always right and it does suck when you can't make an appeal to human compassion. But humans can be just as wrong and just as obstinate, if not moreso. At least with an algorithm you know where you stand.
I, for one, welcome our algorithmic overlords. As a hacker I've always been able to figure out how technical systems work. Other humans? Not so much. If anything, this new world will play to my strengths and to the strengths of those like me.