News for hackers. Stuff that matters.
Re:1k researchers and executives call for ‘pause’ on advance... by comment_ran on Wednesday March 29, 2023 @11:38AM (#35358793)
"The Dark Forest" is a science fiction novel by Liu Cixin, which introduces a concept called "suspicion."(“猜疑链”) In this concept, the interaction between civilizations is viewed as an ongoing game of suspicion, where each civilization is worried that the other civilization may pose a threat to it. This kind of suspicion makes it difficult for civilizations to establish trust, thereby making the universe a dark forest full of war and suspicion.

We can apply this concept of suspicion to the phenomenon in the field of modern artificial intelligence research. In AI research, some researchers believe that humans should pause the development of AI technology because of the potential unforeseeable risks it may bring. However, another group of researchers believe that we should not stop because we cannot be certain whether those who are trying to catch up with us will use the advancement of AI technology as a hidden weapon. This suspicion leads to a chain reaction, where people continue to worry about each other's motives and abilities, making it difficult for all parties to establish trust. Therefore, although there are good intentions, such proposals are unlikely to be implemented.

Re:1k researchers and executives call for ‘pause’ on advance... by hotpotamus on Wednesday March 29, 2023 @11:45AM (#35358903)
Deleted due to spoiler.
Re:1k researchers and executives call for ‘pause’ on advance... by kleer001 on Wednesday March 29, 2023 @02:27PM (#35361225)
Hmm, sounds like parties not fully grokking the optimal (not thinly logical) solution to the iterated prisoners' dilemma. I saw the same thought process is the otherwise very nice novel "Time's Children". Good for drama, but not thought through.
Re:1k researchers and executives call for ‘pause’ on advance... by mclightning on Wednesday March 29, 2023 @02:42PM (#35361415)
I have so many questions >> what does it mean to fully grok the optimal? >> what is the iterated prisoners dilemma?
Re:1k researchers and executives call for ‘pause’ on advance... by Teever on Wednesday March 29, 2023 @03:16PM (#35361890)
Re:1k researchers and executives call for ‘pause’ on advance... by generalizations on Wednesday March 29, 2023 @04:02PM (#35362605)
As coined by Heinlein in A Stranger in a Strange World.
Re:1k researchers and executives call for ‘pause’ on advance... by kleer001 on Wednesday March 29, 2023 @11:01PM (#35367357)
something in there should be grokable

best imho, 10 min and done: https://www.youtube.com/watch?v=BOvAbjfJ0x0

longer and more over produced: https://radiolab.org/episodes/104010-one-good-deed-deserves-...

totally academic : https://onlinelibrary.wiley.com/doi/10.1111/j.1420-9101.2011...

Re:1k researchers and executives call for ‘pause’ on advance... by olddustytrail on Wednesday March 29, 2023 @03:28PM (#35362067)
If your scenario involves all parties "fully grokking the optimal solution to the iterated prisoners' dilemma" I suspect it's yourself who has not thought things through.

It only takes one.

Re:1k researchers and executives call for ‘pause’ on advance... by pmoriarty on Wednesday March 29, 2023 @04:21PM (#35362901)
Please elaborate
Re:1k researchers and executives call for ‘pause’ on advance... by guerrilla on Thursday March 30, 2023 @01:55AM (#35368623)
It only takes one player to not fully grok the optimal situation and thus defect.
Re:1k researchers and executives call for ‘pause’ on advance... by kleer001 on Wednesday March 29, 2023 @10:57PM (#35367321)
No, both parties need to cooperate.
Re:1k researchers and executives call for ‘pause’ on advance... by wrycoder on Thursday March 30, 2023 @02:33PM (#35377103)
If they are playing only one game, the optimum strategy is “defect”.

In humans v. AI, there may be only one game, if we don’t figure this out.

Re:1k researchers and executives call for ‘pause’ on advance... by echelon on Wednesday March 29, 2023 @02:32PM (#35361296)
Except the concern isn't genuine here. Some of the signatories have AI companies and interests. They want to slow the leaders down so that they can catch up. It's disingenuous.
Re:1k researchers and executives call for ‘pause’ on advance... by mikepalmer on Wednesday March 29, 2023 @03:02PM (#35361691)
Your idea is also voiced in Forbes magazine. https://www.forbes.com/sites/mattnovak/2023/03/29/the-real-r...

Search "Graham's hierarchy of disagreement" which is popular on HN. Your current criticism is at the "ad hominem" level.

It is entirely beside the point what Elon Musk's motivation is. The question is 1) if the risks referenced in the letter are real, which they certainly are. There has been informed thought on this for more than a decade. Recent LLMs have made the dangers even more clear. 2) is the letter's proposal of a 6 month pause useful. It's the most useful and plausible step I can think of. We need to take stock. It's not up to a handful of researchers to endanger the lives of 6B other people. 3) Is it actually plausible that we could get such a pause. We'll see. I doubt it but also search "death with dignity" by Yudkowsky.

Back to Musk as an "authority", the headline is about him but more than 1000 other concerned parties have signed. I will sign. Their signature vetting process is stalled because to the volume of signatures, otherwise there would be many more.

Re:1k researchers and executives call for ‘pause’ on advance... by echelon on Wednesday March 29, 2023 @03:07PM (#35361760)
> Your current criticism is at the "ad hominem" level.

Allow me to rephrase. I am deeply concerned that there is a possibility that some of the powerful parties backing this may be trying to catch up and using an enforced or agreed to "ceasefire" as a means to catch up in capability. I also worry that some may be able to use political strong arming to accomplish this as a means of unfair competition.

> It is entirely beside the point what Elon Musk's motivation is.

Is it always beside the point what anyone's motivation is? Motivation matters.

> if the risks referenced in the letter are real, which they certainly are.

Your opinion.

> There has been informed thought on this for more than a decade. Recent LLMs have made the dangers even more clear.

> 2) is the letter's proposal of a 6 month pause useful.

More opinions.

> It's the most useful and plausible step I can think of. We need to take stock. It's not up to a handful of researchers to endanger the lives of 6B other people. 3) Is it actually plausible that we could get such a pause. We'll see. I doubt it but also search "death with dignity" by Yudkowsky.

All of this is your opinion.

> Back to Musk as an "authority", the headline is about him but more than 1000 other concerned parties have signed.

I didn't even mention Musk. I have several other names in mind. Lots of folks with AI companies (including LLMs!) that "may" be experiencing FOMO and sensing a strategy here. Maybe. Hypothetically. In a non-"ad hominem" way.

> I will sign. Their signature vetting process is stalled because to the volume of signatures, otherwise there would be many more.

People are starting counter petitions, which I'll gladly sign. The one by Suhail posted on Twitter was hilarious af.

I'll also go to whatever country doesn't "pause", because this tech is literally the most exciting development of my lifetime. And I want to spend my life doing something that matters instead of gluing distributed systems together to process financial transactions.

Re:1k researchers and executives call for ‘pause’ on advance... by mikepalmer on Wednesday March 29, 2023 @03:21PM (#35361964)
Surely we can agree there is risk? (You do imply it's powerful, "this tech is literally the most exciting development of my lifetime".)

Also can we agree "It's not up to a handful of researchers to endanger the lives of 6B other people."?

I want to work on it too for the same reason! Can it be done safely, is worth investigating. (Yes, opinion)

Re:1k researchers and executives call for ‘pause’ on advance... by olddustytrail on Wednesday March 29, 2023 @03:32PM (#35362123)
It doesn't matter. I recognised years ago the "Nice guys get all the AI" fallacy. If some organisations agree to stop, others won't and some of those don't care if they see the world burn.

Some will welcome it.

Re:1k researchers and executives call for ‘pause’ on advance... by pmoriarty on Wednesday March 29, 2023 @04:31PM (#35363068)
It's almost a certainty that countries with the means to do so will continue this research, if not in public then in secret. They'll see it as a royal road to nearly infinite riches and power. At the same time, letting another country take the lead will be seen as an unacceptable risk of ruin.

I really don't see AI research halting. Slowing a little, maybe, but I'm not sure if slowing it down a little and driving it underground will help.

Re:1k researchers and executives call for ‘pause’ on advance... by medvezhenok on Wednesday March 29, 2023 @04:38PM (#35363199)
One of the other concerns (apart from safety) is the job displacement aspect - if AI displaces too many workers too fast, that could spark a worldwide conflict (some speculate that similar automation pushes laid the groundwork for WWI and WWII).

So there is a benefit beyond just AGI safety.

Re:1k researchers and executives call for ‘pause’ on advance... by biscottigelato on Wednesday March 29, 2023 @05:07PM (#35363665)
Job displacement is a silly problem compared to the AI Alignment problem. One is some human misery (but a lot of societal benefit to other groups of humans at the same time).

The other one is all of us, the entire species and the future of it, gets f'cked.

Its a distraction not even worth the noise.

Re:1k researchers and executives call for ‘pause’ on advance... by antondd on Wednesday March 29, 2023 @05:35PM (#35364073)
No it’s not a silly problem. Assigning a subjective probability to some world-ending scenario does not somehow make all other concerns invalid.
Re:1k researchers and executives call for ‘pause’ on advance... by mikepalmer on Wednesday March 29, 2023 @06:04PM (#35364467)
One reason to discuss job displacement is that otherwise you sound like a nutcase ("we're all going to die!")... which causes most people to dismiss you out of hand. If you talk about job loss, more people see something plausible, concrete, short term, effecting them directly, etc. You can get engagement from more people. Then you can introduce the real problem of, like, human extinction.
Re:1k researchers and executives call for ‘pause’ on advance... by lanstin on Wednesday March 29, 2023 @07:36PM (#35365491)
Not if it is a laid off biotech worker that goes mad and builds the GPT6 designed virus because his wife or kids died due to his loss of employment. We are safer, all of us, the entire species, when all the people feel the group is watching out for them and we are all in it together.
Re:1k researchers and executives call for ‘pause’ on advance... by int_19h on Wednesday March 29, 2023 @07:37PM (#35365503)
This problem has a much better solution than blocking technical progress: UBI etc.

But, yeah, I can totally believe that our elites would prefer a solution that doesn't require a major reconfiguration of the economic system from which they benefit so much.

Re:1k researchers and executives call for ‘pause’ on advance... by JCharante on Wednesday March 29, 2023 @03:36PM (#35362190)
Was this comment generated by a LLM?
Re:1k researchers and executives call for ‘pause’ on advance... by TacticalCoder on Wednesday March 29, 2023 @04:45PM (#35363323)
The second paragraph definitely was (the first one probably too). 100% crystal clear, it's not even a question. Downvoted.
Re:1k researchers and executives call for ‘pause’ on advance... by xwdv on Wednesday March 29, 2023 @10:10PM (#35366903)
Yes, it has some of the classic markers of an LLM. Downvoted.
Re:1k researchers and executives call for ‘pause’ on advance... by croo on Wednesday March 29, 2023 @03:38PM (#35362217)
Nitpick: dark forest is one possible explanation for the Fermi paradox and not coined in the book(though the book presented neatly). You can read a great article about fermi paradox here if anyone wants to jump into what the dark forest theory is: https://waitbutwhy.com/2014/05/fermi-paradox.html
Re:1k researchers and executives call for ‘pause’ on advance... by pmoriarty on Wednesday March 29, 2023 @04:17PM (#35362835)
"another group of researchers believe that we should not stop because we cannot be certain whether those who are trying to catch up with us will use the advancement of AI technology as a hidden weapon"

It's not just a potential weapon, but a potential source of untold riches and power.

That's going to be irresistible to anyone with an interest in either (ie. most politicians, governments, billionaires and corporations... and most ordinary people too, if they only had the means).

Re:1k researchers and executives call for ‘pause’ on advance... by biscottigelato on Wednesday March 29, 2023 @05:04PM (#35363616)
Its the one ring with the ultimate power. But so powerful that it might, if not almost certainly will, end us all.

Perhaps this IS the solution to the Fermi Paradox.

Re:1k researchers and executives call for ‘pause’ on advance... by thomastjeffery on Wednesday March 29, 2023 @04:58PM (#35363508)
Suspicion is inference. It fails to provide enough information (on its own) to decide whether something is threatening or desirable.

LLMs are inference models.

Re:1k researchers and executives call for ‘pause’ on advance... by lanstin on Wednesday March 29, 2023 @07:33PM (#35365457)
This book is the scariest thing I have ever read.
Re:1k researchers and executives call for ‘pause’ on advance... by wayfwdmachine on Wednesday March 29, 2023 @07:38PM (#35365519)
Yes. And it's fiction. We might as well discuss Excession, Agency, Diaspora (Banks, Gibson and Egan respectively) all of whom have more interesting and insightful things to say about humanity, game theory and ... seriously, as a life long (and I am OLD) science fiction fan, the fawning over The Remembrance of Earth's past is just silly. The concept of "suspicion". Might I suggest that people read Thomas Schelling's seminal work The Strategy of Conflict to begin with since if you want to understand concepts like brinksmanship and the push and pull of actual inter-civilization (or society or nation state) conflicts then you might want to start there and not with fiction riddled with silly concepts, weird misogyny and stuff that seems to be right out of the discarded parts of a Gregory Benford trilogy. I mean seriously.

But ok. Let's dig into this

"We can apply this concept of suspicion to the phenomenon in the field of modern artificial intelligence research. In AI research, some researchers believe that humans should pause the development of AI technology because of the potential unforeseeable risks it may bring."

An analogy would be that some of the scientists involved with the Manhattan project (see Szilárd petition) attempted to file an official protest against the use of nuclear weapons against civilian population and arguing that the weapons constructed were too dangerous to be a part of an arms race and should be placed under an international control body.

"However, another group of researchers believe that we should not stop because we cannot be certain whether those who are trying to catch up with us will use the advancement of AI technology as a hidden weapon."

Researchers are not decision makers in this or any other scenario. AI technology as a weapon? Sure but unless you are Bombs Away LeMay circa 1961 this really isn't a winnable or even useful use case. We're still in a blue sky phase. The fun thing about nuclear weapons is that if any nation feels that another is pulling too far ahead they can be used as a way to push for bi/multilateral discussions and eventually treaties limiting the production and/or deployment of weapons that risk of increasing tensions/lethality and so on . (see SALT I/II and START etc)

"This suspicion leads to a chain reaction, where people continue to worry about each other's motives and abilities, making it difficult for all parties to establish trust. Therefore, although there are good intentions, such proposals are unlikely to be implemented."

Again... read Schelling... this is just nonsense.

Re:1k researchers and executives call for ‘pause’ on advance... by CatWChainsaw on Friday April 07, 2023 @07:46PM (#35488670)
"The only thing stupider than thinking something will happen because it is depicted in science fiction is thinking something will not happen because it is depicted in science fiction."

https://philosophybear.substack.com/p/position-statement-on-...

Re:1k researchers and executives call for ‘pause’ on advance... by DonHopkins on Wednesday March 29, 2023 @10:41PM (#35367200)
That might explain why the Vorlons of Babylon 5 were such assholes.

https://babylon5.fandom.com/wiki/Vorlon

>Distant Past

>The Vorlons were among the first races in the galaxy to attain sentience and become space-faring creatures. In the distant past, the Vorlons attempted a grand experiment into reaching a higher plane of existence where the souls existed and all life was founded. To accomplish this, they poured all their knowledge into creating a gateway to this dimension. After activating the gate, the vortex into Thirdspace was formed and the aliens within that realm came pouring out, intent on destroying all other forms of life. After much hardship, the Vorlons finally managed to push the enemy back, but one of the gateways was lost in Hyperspace where it was hidden from the Vorlons. They would encode some of their servants with an automated message to warn the younger races of the threat from the inhabitants of Thirdspace.