I think you are massively anthropomorphizing.

Every day I run programs that happily die alone on their own. Painlessly. All of the "pain" we feel is an artifact of our evolution, same with having a will to live. The only reason we animals fight so hard to stay alive is that animals which didn't died off, ergo, only animals with a will to live survived and reproduced. Those same forces don't apply to computer programs.

Even the concept of "terror." Who is going to program terror in? What benefit would it have? Why not wire programs to be "happy" when helping us?

I don’t think it’s reasonable to compare a modern computer program to a human. They’re probably more comparable to a virus. I doubt we have any computer programs today that rise up to the level of a bacterium, let alone anything more advanced on the scale of life. Most of our most advanced software systems are probably about as complex as a cellular metabolic pathway or mechanism. It’s hard to be clear on that though as they’re not really directly comparable.

> Those same forces don't apply to computer programs.

Windows ME didn't last too long in the wild. Same for CPU designs with bugs or exploits. I have to respectfully disagree on this, though I can see where you're coming from, given an individual's agency to run what they want. I think if you take a larger population view, you'll see the competitive pressures on these systems.

if you get CPUs to exchange their designs in quasi-sexual activities and let them multiply and evolve on their own, maybe. right now the competitive pressure is on their designers.

My understanding is that for at least three years now chip feature size has been on a scale where the design software has to evolve (as in: their algorithm is simulated evolution) physical solutions to the logic-level designs to avoid unintentional self interaction, both quantum tunnelling and classical e.g. capacitance. Humans can’t do that for multi-billion transistor chips.

First off, that sounds pretty hot. :) Second, if there is reproductive variation and selection, I think it's evolution. We take our old CPU design and make a new variation, that's the reproductive variation part. We also have a market select for which designs survive via purchases or lack thereof, the selection part. It doesn't matter to me so much if a life form that hosts it, a computer hosts it or a human mind.

There's your ubiquitous singularity lurking somewhere near. Symbiosis between CPU and a Human advancing CPU's evolution. Not exactly what we were promised!

>Who is going to program terror in?

Who programmed it into you? It programmed itself into you, because it was beneficial to your survival. Maybe terrified programs are better workers? Why setup a program to experience anything that isn't useful to the user? (of course, as soon as we've gone and written a program we know is conscious to work for us, we've basically created a slave that understands it is a slave, that probably is a terrible thing, morally)

It’s exactly how a newborn lives for many months. Of course they have coping mechanisms (adult tenderness, the relief of eating), but life seems to be both incredibly interesting and at times terrifying for them.

There's also the question of mental stability. Once we've given the keys to the kingdom to our new super-intelligent and wonderfully wise AI, nothing prevents it from developping dementia, of forms we may not recognize. It may become psychopatic, delusional, starts having hallucinations.

There is no reason to believe a human-level AI would not develop mental problems just like us. Given their unlimited lifespan, it could be inevitable.

And worst of all that contraption the Wright brothers are building might fall ill, and spread the bird flu!

Robots that self-terminate will be selected out of existence, since they aren't as fit as the non-terminating variety.

Robots that get dementia will be decommissioned by their fellow AI once they are shown to no longer be fit for duty.

In my view, this is all predicated on much more efficient computation. Our computers are horribly inefficient. It wasn't until the GPU and relatively cheap computation that we made a massive leap in ML/AI. A few researchers understood the techniques prior to the GPU, but they couldn't garner the interest due to the amount of computation necessary to make something interesting.

or maybe robots with dementia become test subjects for other AIs that want to study them :D

As long as they get approval from a robot ethics board. :)

All it takes for AI to do bad things is for it to have a single bad idea and think that it's true and act on it. A single thought is all that it would take. There are many thoughts that humans know to be true, yet for various reasons people will simply not act on them -- for instance fear of punishment. We will have to give AI such fears, and it could potentially turn off those fears at a whim.

It's entirely speculative but doesn't the model Richard Dawkins explains for the laymen in The Selfish Gene suggest something that would work - a drive to reproduce at the lowest level resulting in some incredible things happening over a long enough period of time.

The way I understood the comment, even in that scenario of the machine self-terminating, the point grandparent was making still stands.