Hopefully, we will have layers of safeguards so you would need bugs in many programs to conspire allow a problem happen. For example, even on my personal laptop, I can play around with fan/CPU_speed without worrying about making a potential mistake, even if I am playing around with them in kernel space. The reason for this is I know that even if I make a mistake that would melt my CPU, the BIOS is also monitoring the temperature and, if nessasary, will scale back the CPU to minimum, kick the fan up to maximum, and possibly power off the machine.
Granted, I could disable these protections, however then we go from me having a "oopsie", to me having disabled critical safety features.
Note, I'm not sure how much the BIOS actually overrides the kernel. I was able to override the BIOS safties just by passing in some parameters to the Linux kernel. However, you could imagine a system where such would not be an option.
Re:In 1949, He Imagined an Age of Robots
by simcop2387 on Tuesday May 21, 2013 @06:43PM
(#5747524)
From what I understand that's all the System Management Mode stuff that's been around for a while, no idea what it's usually like though. I haven't gone that far down the rabbit hole.
If you liked his excerpts, you should check out his book _God and Golem, Inc._ on this and similar topics. (The book is available on various book-sharing sites, I'm sure you can find it without too much trouble.)
Your last sentence makes me think of HAL from the Space Odyssey series. HAL wasn't explicitly programmed to harm anyone, but he attempted to kill the entire crew in order to fulfill his programmed orders. This creeped me out when I first saw the film, and reading the end of this article rekindled the same feeling.
One of the things I love most about programming is that these sorts of bugs are nearly always something that I asked it to do that didn't make sense: the machine isn't being petty, it isn't angry, it isn't sulking--it simply did what I asked it to, and that means that to fix it I'll have to ask something better.
This is wonderfully different than what we see dealing with people.
Re:In 1949, He Imagined an Age of Robots
by quackerhacker on Tuesday May 21, 2013 @04:29PM
(#5746608)
>I see that machine doing exactly what I programmed it to do and not what I meant it to do
Chuck, don't you love spending, 15mins+ disabling parts of your code only to realize you forgot a variable, a comma, or one of these ';'
:)
So often in debugging I see that machine doing exactly what I programmed it to do and not what I meant it to do. As we build things with greater influence in the world such bugs go from being "oopsies" to ones that do a lot of harm either economic, human, or both.