The concern is that the most informed names, and those spearheading the publicity around these letters, are the most conflicted.
Also, you can't scan bio lines for the affiliations that impact this kind of statement. I'm not disputing that there are honest reasons for concern, but besides job titles there are sponsorships, friendships, self publicity, and a hundred other reasons for smart, "politically sophisticated" people to look the other way on the fact that this statement will be used as a lobbying tool.
Almost everyone, certainly including myself, can agree that there should be active dialog about AI dangers. The dialog is happening! But by failing to make specifics or suggestions (in order to widen the tentpole and avoid the embarrassment of the last letter), they have produced an artifact of generalized fear, which can and will be used by opportunists of all stripes.
Signatories should consider that they are empowering SOMEBODY, but most will have little say in who that is.
Both of them are criticizing their own life's work and the source of their prestige. That has to be emotionally painful. They aren't doing it for fun.
I totally understand not agreeing with AI x-risk concerns on an object level, but I find the casual dismissal bizarre.
If instead you want to consider the highest status/most famous people working on AI in general, then the list of signatories here is a pretty good summary. From my flawed perspective as a casual AI enthusiast, Yann LeCun and Jürgen Schmidhuber are the most glaring omissions (and both have publicly stated their lack of concern about AI x-risk).
Of course, the highest status people aren't necessarily the most relevant people. Unfortunately, it's more difficult for me to judge relevance than fame.
There are people studying how previous societies got into existential risk situations, too.
We also have a huge amount of socio-economic modelling going into climate change, for example.
So I'd say there should be quite a few around.
I also find it somewhat telling that something like "massive wealth disparity" or "massive unemployment" are not on the list, when this is a surefire way to create a highly unstable society and a far more immediate risk than AI going rogue. Risk #5 (below) sort of alludes to it, but misses the mark by pointing towards a hypothetical "regime" instead of companies like OpenAI.
> Value Lock-In
> Highly competent systems could give small groups of people a tremendous amount of power, leading to a lock-in of oppressive systems.
> AI imbued with particular values may determine the values that are propagated into the future. Some argue that the exponentially increasing compute and data barriers to entry make AI a centralizing force. As time progresses, the most powerful AI systems may be designed by and available to fewer and fewer stakeholders. This may enable, for instance, regimes to enforce narrow values through pervasive surveillance and oppressive censorship. Overcoming such a regime could be unlikely, especially if we come to depend on it. Even if creators of these systems know their systems are self-serving or harmful to others, they may have incentives to reinforce their power and avoid distributing control.
Anyone upvoting this comment should take a long look at the names on this letter and realize that many are not conflicted.
Many signers of this letter are more politically sophisticated than the average HN commenter, also. So sure, maybe they're getting rolled by marketers. But also, maybe you're getting rolled by suspicion or bias against the claim they're making.