The more likely AGI risk is indifference in pursuit of optimizing a reward function with values not aligned with us.

Like the anthill destroyed by the construction of a dam. The humans don’t hate the ants, aren’t putting them in zoos, they just don’t think of them in pursuit of their goals. In the AGI case it’s even possible for those goals to be “dumb” or pointless if we mess up the reward function or some unanticipated thing fits it (paper clips).

The risk with a self-improving AGI is that it is way smarter than us and it can think way faster. In my experience people have a hard time really visualizing this.

Imagine a human trying to trick a chimp (or an ant) into doing something so the human can pursue its goal. The gap between a human and an AGI would be even larger than that.

Imagine scaling up a regular human brain’s operations per second a billion times so it’s the same architecture but thinks that much faster (no longer constrained by bio energy or head size). Humans wouldn’t be able to respond because to the machine it’d be nearly like interacting with a static object, it’d be trivially able to think/act around you.

That’s why the research around AGI is about the goal alignment problem. Which is hard because humans are not perfectly aligned either. It’s also why “just unplug it” is dumb.

The optimistic outcome if the alignment problem is solved is enormous intellectual capacity to solve any problem we might have.

The more likely AGI risk is that we expend tremendous resources trying to develop it, only to find it doesn't work very well or do anything useful.

There’s nothing that makes it physically impossible so it’s likely inevitable - just hard to know when.

The risk of poorly aligned AGI as described is real, makes sense to try to work on the problem before we need it (when it’ll be too late).

No it's not a real risk, just a silly fantasy with no basis in scientific reality.

Dismissive comments without any substance make you come across as dumb, not smart.

If you have some actual position of substance, articulate it or just don’t bother replying at all.

Claims about AGI without hard scientific backing make you come across as pseudo-intellectual, not smart. If you have actual evidence then articulate it or just don't bother replying at all.

Claims made without evidence can be dismissed without evidence.

If AGI is possible, then the described concerns about risk hold up.

Either you think AGI is impossible for some reason or you think it is possible, but the goal alignment issues don’t matter. You haven’t given any explanation for either. Maybe you think something else? I wouldn’t know because you don’t say anything.

Obviously there isn’t “hard scientific backing” since AGI doesn’t exist yet. There are still unknowns about what would be required. That doesn’t mean it’s impossible to think about what its existence could mean.

It was possible to think about the consequences of flight before airplanes existed.

I think AGI is possible because we’re surrounded by general intelligence and there’s nothing magical about brains or biology. It’s just the timeframe that’s unclear.

We already see issues with poorly optimized reward functions on a smaller scale, both in ML models (classifying based on unintended things) and in humans (drug addiction).

So in other words you're just engaging in idle speculation.

No - it logically follows and you can begin work on the problem now, but it's obvious you refuse engage in anything other than one line dismissals and condescending snark so there's no point in me replying further.

Logic only produces correct results when your starting assumptions are correct. Since you're just making things up, and logic resting on that foundation is pointless.

Ted Chiang believes it is AIs that have much to fear from us, not the other way around. He points out that to create intelligence, we will need to create something that is capable of suffering. He asks us to consider how little consideration we give to the billions of animals trapped in our system of industrial farming - animals we know are intelligent, feel emotions and are clearly capable of suffering - and then asks if we think we will treat computer programs with more consideration than that. Clearly not!

The correct alternative path is to just not do it. Let’s not attempt to create intelligent beings. Given the suffering we would cause, doing so is profoundly unethical.

I love Ted Chiang’s scifi but he’s just not well informed on the AGI stuff. I think if he ramped up on it his writing about it would better.

While I have great compassion for anti-nativism, in the past few decades we have made immense sociological progress. If it is possible to create life that is incapable of shame or suffering and yet capable of enjoying its existence for its own sake, we should try to find out about it before turning the lights off.

I always thought that people are afraid of AGI for the very reason of knowing how they treat beings of lesser intelligence, hence the popularity of movies like the matrix - because that is what we'd (presumably) do if we were in their position.

I suppose this also extends to having children?

Yes but they get to abuse their own children so we are "even".

Please don't take this seriously. A lot of hazing culture is based around the fact that you got bullied and now you get your revenge by bullying someone else. We shouldn't perpetuate that.

Any alien with the ability to physically reach earth either via FTL or other dimensions, etc. is going to harvest our planet/solar-system for resources, they aren't going to give a damn about us.

Not unlike maybe Star Trek Discovery latest season but on a lesser scale/tech.

I never understand people over-evaluating our interest to something else exponentially more advanced. We wouldn't be like farm animals for labor, their tech/machines would be exponentially more efficient.

This is also why I think UFO believers are not rational. Why would they even care, we'd be annoying at best.

It seems to me that we are the resources of this planet. If you just desired materials, any number of objects in space would be better suited. Unless it's the goldilocks zone of environments that you are after, a habitable planet might be harder to come by.

Haha! So true. But even if we aren’t as intelligent as them, we could be “unique”. Someone would probably want a collection of this “cute” but less intelligent humans/animals?

Thank god we are first (vs animal life).