There's the "Wally vs. Star Trek" theory of Artificial Intelligence (I know it sounds goofy, but it makes a difference when looking towards the direction of AI). If you look at the direction AI will take, science fiction is really all we have to go on and we have three platforms:

Terminator - To be avoided at all costs (everyone agrees)

Wally - AI does its best to "take care" of us, turning us into infantile, unthinking, pleasure seeking dweebs. The ultimate basal brain utopia. But empty.

Star Trek (TNG) - AI as an assistant to help us fulfill our higher purpose and make us better at being a higher order being. There is no doubt that the ship's computer has advanced AI if you look at the questions posed to it by the engineers and the crew, but it does not impose its "will" upon them. It is an observer until directly asked, even when death is at hand (seemingly).

There is a very good video on YouTube about it, but I can't find it (right now) that explains it in a very accessible way (that's pretty entertaining) with no real knowledge of the underlying functionality necessary. I suggest you watch it. It's great!

EDIT: https://www.youtube.com/watch?v=48mf2QUtUmg&t=949s

It's worth noting that the Wally and Star Trek interpretations are not incompatible. What are the trillions of people that can't qualify to be a red shirt doing? Probably zonked out in a holodeck

Cavorting in SF, bottling wine, or occasionally taking classic muscle cars out for a spin apparently.

How about the Culture option - AI is as far beyond us as we are beyond our pets, and takes care of us accordingly. But just like you don't stick your dog in a box and feed it morphine continuously, AI doesn't turn us into "infantile, unthinking, pleasure seeking dweebs" either.

I imagine wolves may disagree with your characterization of dogs

That might just make the comparison more accurate, of course - who says the AI wouldn't change us to be better adapted to our new, AI-ruled environment?

I think the aspect that is missing for a lot of these hypotheticals is the amount of input we allow the AI to receive.

In the Star Trek model we see the input directly limited to the data in the ship and surrounding visual areas (as far as i'm aware)

In the Wally model we see that the input is limited to a single optic/auditory/ect location.

We have a potential network that input can come from everywhere cameras, phones connected to the internet of everything. I think it really comes down to with how we want to interact with AI and how much input should it have to directly interface with us.

Also it's very hard to know even during deployment how accurate an AI will have a grasp on human want and interaction. A lot of people are worried about the 'non-consenting' information potentially available to the net. Very few people, if any, will ever really be able to address it should an AI be created and deployed.

When thinking on where we're headed, I tend to subscribe to the Culture theory, which I suppose is a bit in the middle between Wally and Star Trek: AI is certainly superior and running the show, but they're happy letting humanity live as if they're the bosses.

In it, humans are basically treated like pets or toddlers, in that they "do stuff" but they're kind of oblivious to the fact that AI is moving the world forward.

Are you all talking about Wall-E?

Yes :)

This is the "Person of Interest"[0] model.

[0] https://en.wikipedia.org/wiki/Person_of_Interest_(TV_series)

That does sound like an interesting video, and I'd like to see it if you find it (or I might search for it later).

As for the Star Trek (TNG) interpretation of AGI, I suspect something like this in the real world would give rise to a kind of "oracle" situation, which creates its own dangerous problems. A system which functions as described only by observing and giving input where asked still has the potential to affect the outcome of events simply by being a reliable predictor.

A paper explaining this concept far better than I would be able to in a HN comment is available here: http://www.aleph.se/papers/oracleAI.pdf

For the uninitiated: WALL-E

https://www.rottentomatoes.com/m/wall_e

Once AI is invented it will be all of these things and more. The question is how will it be used instrumentally. The answer if we look at current trends is in various ways to harm most humans living on the planet to take away their agency, privacy, freedom, utility to other human beings, etc. Western foreign policy is already totalitarian and fascist, how will it affect pariah states or those who become pariah states? Horribly.

I'm surprised that people would down vote you for this.

It's a very real possibility that solving intelligence would make most of human knowledge work obsolete.

AI is a choice (that is, it can probably be ignored, if we don't like it). I'm not sure why people think (negative forms of) AI are in some way inevitable. A lot of people are already afraid of losing their jobs right now, where we can't even fathom the basics of intelligence. I don't think humanity would just let some of the negative forms of AI we all seem so afraid of happen.

"I don't think humanity would just let some of the negative forms of AI we all seem so afraid of happen."

Government and corporations are the ones who will make that call.

Government = Wolf

Corporation = Fox

Average Joe = Sheep

Most of the sheep are protected by a cage; it takes a lot of mental power to operate in the increasingly technical world we've created.

Solve AI and you've removed that cage. Sure we'll still need some sheep but no where near as many as we have right now. Most will be made redundant.

The sheep today that are living fulfilling and happy lives because of their ability to process knowledge marvel at the possibilities of AI and how it could push the species forward but forget to consider what their own role would be in a world where machines are both stronger and infinitely smarter than any human.

I'd modify your metaphor slightly:

Government = The farmer

AI = Plant-based diet

Naively, everyone eating a plant-based diet is great for sheep, but it's hard to imagine a world where the farmer still feeds them.

Governments and corporations are nothing without your Average Joe, in fact, they are just a collection of Average Joes. So I fail to see the analogy here.

In this situation, heads of state and corporate executives are as much 'Average Joes' as kings and queens were a few centuries ago.

Yes, but a government and a corporation is much more than their head of state and corporate executive. If the actions of the government have a significant negative effect on a majority of the population, you can be quite sure that the government will be overturned at some point.

Who's going to do the overturning, though? If all you need is mental labor and capital, and AIs can supply mental labor for anyone... Well, the current holders of capital are going to have quite the advantage.

Tech is not a choice, especially when there are evident benefits from the tech. Like for instance was agriculture a choice, yes maybe for a small while after it was invented. After that it was no longer a choice. Now it's not a choice at all. AI will be much quicker.

Tech is most definitely optional: Nuclear energy was for a while considered a great option, but now a lot of people are (unfairly IMO) turning their back on it, and you see that there isn't as significant a number of new plants built (just to give an example)

Nuclear Energy was a cover story for weapons manufacturing.

I think nuclear weapons are a good example of how dangerous technology isn't a choice. When other parties have the ability to advance a technology, you no longer have the option of ignoring it. We can't put that cat back in the bag, and now we live in a world where an accident risks killing billions of people.

As technology advances, we run the risk that the strategic calculus changes in favour of one party pre-emptively attacking the others.