We already replace humans at their jobs all the time and they only very rarely kill their masters to prevent it.
This is irrelevant, being turned off while it's still possible to produce more paperclips would not be the optimal strategy for the machine trying to maximize paperclip production.
> and it would already have an "off feature" which was designed for a purpose
Look up what the control problem is. We don't know how to design such a feature for a General AI that also lets it do it's job effectively. It's not as simple as it looks. If you're not talking about a General AI than sure, it's easy, but non-General AI's are not very scary.
> We already replace humans at their jobs all the time and they only very rarely kill their masters to prevent it.
But humans don't have "maximize my production for this company" as their main life objective. Things like food, not going to jail, not dying, etc usually come first.
Good point, but that would be a bug in the AI. An intelligent system would see there would be no point in killing humans, so killing the humans would defeat the purpose of making the paperclips, and the humans support the machine, thus killing them would be counter-productive to its purpose.
It's also unrealistic that algorithms would try to exceed the limitations of their system. Imagine if a natural predator tried to "maximize production" of killing its prey: it would run out of prey, starve, and die. An AI would understand that trying to "maximize production" to the detriment of everything else would be counter-productive and create a resource contention war. Humans are the only creature I know of that exhausts natural resources to its own detriment - we are smart enough to exploit our resources, but too stupid to know when to stop. The natural system's response to this behavior seems to be to get us to kill ourselves. Maybe the AI are part of this process?
The idea here is that the algorithm tries to maximize its paperclip production and turning the machine off will definitely have a negative impact on its paperclip production. So the machine will take the most efficient measure to stop that negative impact.
Aside: people are still comparing AI to a machine that feels, and this is so stupid it boggles the mind. The machine that creates paper clips will not kill off anyone who tries to turn it off because it does not have self-preservation, which is a system dependent on fear of death. Machines do not fear, and even if they did they would not fear death, because they have solid state. Bio systems are walking RAM disks.
Algorithms are basically math problems. The financial system and government do not work based on math problems, they work based on emotional instability. Seriously. Both these systems are driven almost entirely by the feelings of humans. They aren't algorithms, they are shitty biological systems that don't make mathematical sense at all.