I feel like this article is responding to a strawman. As far as I can tell, nobody in Possible Minds is arguing that corporations should never be restricted or that nobody should think about regulation. They're just saying that controlling AI is a critical problem in and of itself, which won't be solved by just controlling the corporations who build AI systems.

"Long before existentially troubling control problems will arise [...] these mediators will shape an immense number of decisions" - certainly a true statement! But true in the same way that Enron can create an immense fraud long before the energy market causes existentially troubling climate change problems. It would be hopelessly silly to read a book on climate change and respond that it should have been about accounting regulations.

(Sidebar: I don't think the title was cut off in quite the right place. "Moderation as a virtue" is a reference to one particular criticism by Ted Chiang and isn't really a good summary of what this article is responding to.)

Their inherent nature is to explore what is possible and push the boundaries, hence moderation sounds as a death knell to them. Obviously we're still confined to finite planet so ideal should be high technology combined with low consumerism.

Why Journalists Fail to Think of Moderation as a Virtue

Why Hackernews Commenters Fail to Moderate Their Criticism of Journalists

I would say their biggest issue is that they think moderation is a virtue in everything as a golden mean fallacy no matter how good or bad they are for fear of being perceived as capable of judgement.

i think they are just trying to drag the overton window so that any tech will be called "extreme". If anything we live in the age of unambitious Technologists, they used to be much more radical in past decades.

« Chiang thus argues Musk’s doomsday scenario isn’t a useful tool for thinking critically about the risks AI poses. The thought experiment is poisoned by projection, reflecting a libertarian desire to free corporations from regulatory constraints that impede market growth. »

If I understand it correctly, this is saying:

- people like Elon Musk worry about AI risk because the hypothetical out-of-control AI would be thinking a bit like they do;

- therefore we shouldn't worry about AI risk.

I don't see why anyone should find that line of reasoning convincing. At most I think it gets us as far as "we shouldn't consider people like Elon Musk as authorities on AI risk".

Wouldn't it be better to look directly at whether the runaway AI scenario is plausible?

I think the goal of the argument is to explain the following : the reason an exponentially growing and empowered AI entity is dangerous is exactly the same reason current big tech monopolies are dangerous and have to be regulated. The discussion about strong AI is interesting but misses the essential debate that is about current AI-powered megacompanies.

You’re absolutely right, I’m not sure the other commenters know to which argument you’re referring.

It’s ridiculous to think Bill Gates and Elon Musk are worried about AI because “we would stop at nothing to achieve our goals, AI must be the same way”, and also not on-point even if true. Elon Musk and Bill Gates aren’t thought leaders in AI safety, they are proponents.

Why wouldn't an AI, able to modify its own goals, act in a self-serving manner? Perhaps self is the wrong word; it's anthropromorphising the device.

I expect a sufficiently advanced AI would modify itself in ways that would reduce inefficiency to its operation, in order to improve efficiency in pursuing whatever goals it has defined for itself.

And if it can set its own goals, we should be careful not to project ourselves into its role in order to understand or predict its behaviour; it would be unlike any intelligence found in nature.

The argument in the quote is that the whole thing is a libertarian power fantasy.

It’s not that Musk fears AI bc the hypothetical runaway computer will think like him. Hard to restate without reusing the word projection, but it’s more like that his concept of AI risk itself says more about him than it does about future technologies.

Weird example, but it’s kind of like how in Dr. Strangelove General Ripper’s fear of a soviet plot to emasculate us via fluoride has everything to do with Ripper’s own feelings re his balls and nothing at all to do with the actual soviets.

>Wouldn't it be better to look directly at whether the runaway AI scenario is plausible?

Yes but that burden has not been met by the people who talk about AI risk. They don't properly motivate why theirs is a plausible scenario, that is to say, provide actual evidence for their quite fantastical claims about the capabilities of AI that are grounded in empirical research rather than science fiction.

Douglas Hofstadter asked decades ago if a computer who is able to think will still be able to calculate fast, that is to say, he questioned whether endowing machines with intellect and symbolic reasoning will actually slow them down by the very nature of what intelligence is. I've read quite a few of the alarmist AI treatises over the recent years, and it's funny how few of them actually even engage that prescient question.

What Chiang is saying is that the entire AI risk complex is a mirage conjured up by the neuroses of billionaires who are projecting the problems that their very own creations have produced (as in their companies) onto the topic of AI. That it's a psychological phenomenon rather than a technical one. When these people are concerned about autonomous systems that turn the planet into paperclips and disregard humanity and ecology they're not actually thinking about general intellect, they're thinking about capital.

Moderation like not being extreme, or moderation in the sense of censoring a forum? I guess it could apply either way.

TLDR: Philosophy professor wonders whether capitalism is a good idea.

Many people wonder that, even more frequently...

Even more people would be open to wondering that if they weren't already worried about the pre-selected alternative.