So.. have we become used to the “computer software” bit or the “bringing us all down” bit?

Not Yanis Varoufakis. His point (not just his of course) is that our conventional software stacks have already mutated society and capitalism. Huge concern about entering into what he calls a technofeudal society in which we are serfs to for-profit platforms. We provide the platform both its content (our wishes and desires) and it then reaps the profits of our purchases and rentals.

So AI and then AGI—just the next mutation two predictable algorithmic and societal pandemic supermutations.

Welcome to the singularity.

youtube.com/watch?v=1A4dMK7S6KE

Sorry are you equating ai and software?

When you look at some standard AI textbook, such as Russel/Norvig, you see that there is not much about being called „AI“. The simplest „intelligent agents“ are functions with an „if“ statement. The smallest Node.js application has more complexity.

AI is software and AI is a term as broad and unspecific as "software".

Software is the purely informational elements and constructs of a computing mechanism.

Or

(more broadly) Software is any construct that is functionally equivalent to its description.

(edit)

It's a useful tool when examining the impact on moral questions, so much of the talk about the transformative power of AI becomes more clear you give up the pretence that introducing AI creates a new class of moral actor that breaks the conventional chains of responsibility.

A recent example of how people try to use this mystical power of AI to absolve you of responsibility of your actions is how UnitedHealthcare, an organisation largely in the business of suppressing health care to those in need, introduced an atrociously bad "AI" to help them deny applications for coverage.

In that example it is very clear that the "AI" is simply a inert tool used by UHC leadership to provide the pretext they feel is needed to force the line workers to deny more care without the whole thing blowing up because of moral objections.

You really need to abandon the "just keep it in the box" argument (i.e., "why give it such a level of control"). It's already out of the box.

Even if it weren't it could trivially convince or fool one of the hundreds/thousands of people working on it to do what it wants. In light of recent events, it is not reassuring to think that the people developing the next generations of AI have both the will and insight to even know where the box is exactly.

A motivated individual could repeat the effort on city power, diesel supplement, a mishmash of cpus cobbled together to few thousands of nodes and a cleverly selected training set... And then connect it to the Internet on purpose. Frankly I think the eventuality of this are llm created firewalls of viruses consuming ever increasing amounts of energy to destroy any uninitiated/unpaying connector. Across the world knots will form at the only borders that matter anymore. Eventually one ID provider will protect us all from noone and the only people who need to worry are those who incidentally wind up on the wrong side of everywhere.

> There's questions that begs to be asked- if we don't trust such "ai" (ie. software), why give it such level of control at all?

You said it yourself: because of our limitless stupidity.

If AI is only slightly less stupid than we are, that doesn't bode well for us.