News for hackers. Stuff that matters.
Re:In 1949, He Imagined an Age of Robots by ChuckMcM on Tuesday May 21, 2013 @04:09PM (#5746510)
I would have loved to talk to this person. I was particularly struck by this comment: Finally the machines will do what we ask them to do and not what we ought to ask them to do. In the discussion of the relation between man and powerful agencies controlled by man, the gnomic wisdom of the folk tales has a value far beyond the books of our sociologists.

So often in debugging I see that machine doing exactly what I programmed it to do and not what I meant it to do. As we build things with greater influence in the world such bugs go from being "oopsies" to ones that do a lot of harm either economic, human, or both.

Re:In 1949, He Imagined an Age of Robots by gizmo686 on Tuesday May 21, 2013 @06:10PM (#5747274)
Hopefully, we will have layers of safeguards so you would need bugs in many programs to conspire allow a problem happen. For example, even on my personal laptop, I can play around with fan/CPU_speed without worrying about making a potential mistake, even if I am playing around with them in kernel space. The reason for this is I know that even if I make a mistake that would melt my CPU, the BIOS is also monitoring the temperature and, if nessasary, will scale back the CPU to minimum, kick the fan up to maximum, and possibly power off the machine.

Granted, I could disable these protections, however then we go from me having a "oopsie", to me having disabled critical safety features.

Note, I'm not sure how much the BIOS actually overrides the kernel. I was able to override the BIOS safties just by passing in some parameters to the Linux kernel. However, you could imagine a system where such would not be an option.

Re:In 1949, He Imagined an Age of Robots by simcop2387 on Tuesday May 21, 2013 @06:43PM (#5747524)
From what I understand that's all the System Management Mode stuff that's been around for a while, no idea what it's usually like though. I haven't gone that far down the rabbit hole.

http://en.wikipedia.org/wiki/System_Management_Mode

Re:In 1949, He Imagined an Age of Robots by gwern on Tuesday May 21, 2013 @10:59PM (#5748597)
If you liked his excerpts, you should check out his book _God and Golem, Inc._ on this and similar topics. (The book is available on various book-sharing sites, I'm sure you can find it without too much trouble.)
Re:In 1949, He Imagined an Age of Robots by fjk on Tuesday May 21, 2013 @04:38PM (#5746664)
Your last sentence makes me think of HAL from the Space Odyssey series. HAL wasn't explicitly programmed to harm anyone, but he attempted to kill the entire crew in order to fulfill his programmed orders. This creeped me out when I first saw the film, and reading the end of this article rekindled the same feeling.
Re:In 1949, He Imagined an Age of Robots by angersock on Tuesday May 21, 2013 @04:34PM (#5746642)
One of the things I love most about programming is that these sorts of bugs are nearly always something that I asked it to do that didn't make sense: the machine isn't being petty, it isn't angry, it isn't sulking--it simply did what I asked it to, and that means that to fix it I'll have to ask something better.

This is wonderfully different than what we see dealing with people.

Re:In 1949, He Imagined an Age of Robots by quackerhacker on Tuesday May 21, 2013 @04:29PM (#5746608)
>I see that machine doing exactly what I programmed it to do and not what I meant it to do

Chuck, don't you love spending, 15mins+ disabling parts of your code only to realize you forgot a variable, a comma, or one of these ';'

:)

Re:In 1949, He Imagined an Age of Robots by ChuckMcM on Tuesday May 21, 2013 @09:29PM (#5748236)
Heh, yet another reason I don't write HFT algorithms, ever.