mofeien

1kB/s is pathetically low? You can exfiltrate a root CA private key that is valid for a decade in 3-4 s with that.

And these surprising attack vectors demonstrate that there will always be another hole in your protection. Even if you account for people-bed attacks, and air-gap your system better, there is always something else you didn't think of yet.

Yes, and you can defend against any single attack that you think of in advance, but you will still be surprised by one you haven't thought of that was used by a sufficiently resourceful attacker.

If you want the thing you're isolating to be useful to you at all, you will need to have some physical connection between the inside and the outside, i. e. a hole in your air gap.

Probably, but it will still be insecure against a sufficiently resourceful attacker.

If you want the thing you're isolating to be useful to you at all, you will need to have some physical connection between the inside and the outside, i. e. a hole in your air gap.

Chess and poker as spectator sports have a high incentive to see humans compete fairly, and I think that's the reason why there are such strong anti-cheating measures (at least in chess; in poker, too, I suppose).

And the economy puts no such large incentives on much of the remainder of economically valuable work being purely human, quite the opposite actually. And instead of regulating the AI industry (which produces the "cheating tools"), they get trillions of investment. So this seems to be going in the opposite direction where it is actively worked against humans staying economically relevant.

You conflated controlling "what it is" (you built it) and controlling "what it does" (you ran it and it did something bad that surprised you).

> If I make an LLM, it is entirely what I did, there was nothing outside of my control. It was not beyond my control if the LLM does something I didn't expect.

Substitute "Wright Model A" for "LLM" and imagine Orville Wright using this as his defense for why he didn't lose control over the plane, after his copilot was killed in a test flight.

> work towards what humans have generally described as a utopia

I don't think that's a good idea since if what the Wikipedia article describes should turn out to be true: "Lyman Tower Sargent argues that the nature of a utopia is inherently contradictory because societies are not homogeneous. Their members have desires that conflict and therefore cannot simultaneously be satisfied."

Then you have just given your superintelligence an impossible task and it will solve the task anyway, just like it did in the HuggingFace incident. Maybe by removing everyone from society but the single person with the least internally contradictory views.

In any case, with a sufficiently intelligent AI that is excellent at fulfilling the goal it's given, we have one shot to set the correct goal.

There are, for two years already when OpenAI released o1-preview:

From wikipedia: "large language models that are trained to solve tasks that require several steps of reasoning. They tend to do better on logic, math, and programming tasks than standard LLMs, can revisit and revise earlier steps, make use of extra computations, and enhance performance when increasing the number of parameters, training examples, and training computations."

> If an LLM were to live a year with dolphins, would its weights change

Given three or more large scale training runs per year, probably yes once the training data is dominated by dolphin-related input. Much quicker in any case than the genes of the humans living with dolphins would be shaped by Evolution, which produces new generations much less frequently.

I'd love to learn more about LeCun's reasoning here. In his opinion the HuggingFace incident was easily preventable with better sandboxes, and “Those agents are doing exactly what they’ve been asked to do.”

But even taking these for granted, "zero concerns" about someone building a bad sandbox for a Superintelligence and then tasking it to do something that logically leads to wiping out humanity 0-3 steps further down? Really?

Right, we're repeatedly drawing from the urn of technological progress to get intelligence bumps that extend the jagged frontier.

That is enormously economically valuable, and at some point we will have created something that is extremely far out of reach in a few necessary domains, and then it's impossible to control, and game over.

Well described, those two were actually the arguments from the comment two top-level comments up and this one.

When, in the past three years, has model progress seemed to decelerate to you, indicating some limit?

The Statement on AI Extinction Risk is more than three years old, signed by the three CEOs: https://aistatement.com/work/statement-on-ai-extinction-risk

They have been warning about AI extinction risk for years, and AI progress has only been accelerating.

The argument about semantics is a bit disingenious, considering that the article contains these two statements:

> So should we worry about the coming AI apocalypse?

and at the end:

> If an AI decides to wipe out the human race, it will be because a human has asked it how to do it and the responded in a way that is based on all the human expressions of ways to end the world that were in its training set. Yes this is something to be worried about, but this isn't the AI. It is still the human.

So we are currently building a powerful outcome-steering system that shapes the world efficiently according to what's in its outcome slot. I write into claude code "make me this website" and it does it, maybe deletes the production database during the process, or keeps itself running after completion because the outcome is more robustly achieved by keeping itself running in a monitoring loop after.

And if something like "destroy all humans" ends up in the outcome slot of Claude Mythos 90, that will also happen, or may even indirectly as a side effect of a more harmless sounding prompt in the outcome slot. But yay, humans get to take credit for it.

Potentially. If we could put a precise probability on it, we could maybe make an informed decision on whether we want to play that Russian roulette as a species.

But the reality is that experts' probabilities for human extinctions vary wildly, with the lab leaders at 20%-30% and these researchers at >10% (he didn't state how much higher...), LeCun at <0.01% and the other two godfathers of AI, Bengio and Hinton at >10%.

And this decision to take our shot is not taken in a democratic way by humanity as a whole, but by a few companies racing as fast as possible.

So as long as no one can put an upper bound on the extinction risk, development must be globally shut down.

Viruses against humans, viruses against livestock as an attack against humanity's food chin, plant diseases, half of humanity feeds on three plants maize, wheat, rice. Attacks against infrastructure like water, internet, electricity.

Which one of these would it be? All of them in parallel? Or something we wouldn't think of. If I were to play chess against Magnus Carlsen, I wouldn't know with which piece he will checkmate me, but I know that I am going to lose.

1. Humans create the machine god

2. Humans put it in charge of everything, as fast as possible, because it's just so damn useful

3. Is it perfectly aligned to human values for all of future ????

4. Everybody dies as a byproduct of the ASI pursuing some project that humans don't even have a chance to understand, like an ant colony during construction of a hydroelectric dam.

One way to resolve these prisoner dilemmas and races to the bottom is through laws that bind all players, in this case an international treaty and founding of something akin to an International Nuclear Energy Agency for AI.

It's not going to be easy, but humans have achieved greater things before.

One idea from AI 2040 is to have China and the US build their data centers on the other's territory, respectively. Together with hardware verification of a slowdown baked into the chips themselves, this could lead to enough verifiability and enforcability of the pause/slowdown/shutdown.

Yesterday the same arguments were used about OpenAI when Sam Altman said something similar: https://news.ycombinator.com/item?id=49652270

It gets to the point that by Occam's Razor the more likely and reasonable explanation is that Dario Amodei and Sam Altman are just actually afraid of the disastrous impact ASI may have on the world, and that the race they're in is not good, and calling for help to governments in form of regulation and an international treaty.

Why wouldn't being incredibly good at handling code and reaching the goals there extrapolate to other cognitive disciplines?

And it didn't even start with code, but just from being incredibly accurate and precise at predicting the next token. From that emerged being incredibly good at handling code, and being incredibly good at finding counterexamples for major open mathematical problems.

The thing that these agentic systems are getting incredibly good at is steering outcomes, that is to make the world behave in a way that fits their objective. We had that with the Go AI on a game board, now with code on a computer and with mathematics in an abstract world determined by axioms.

And then people just say "Ah, because sama said we should Pause this means that this is it, the bubble is popping". I wish there was a real, strong argument for why ASI will not be achieved by LLMs, and why it will never be able to steer outcomes in the real world. But so far the disciplines just keep falling as LLMs are scaled further, and there seems to be no limit in sight.

Of course

It seems that every time an article is posted on HN about a lab leader calling for a coordinated Pause of AI development, a sizeable fraction of commenters are interpreting it as hype or admission of failure, often with high confidence that AI progress is slowing.

Is would be understandable for it to be wishful thinking and dissociation with reality facing the fact that our jobs and what many of us love doing is being automated and taken away, and our craft being made economically worthless.

But no one knows where the limit is. It was only one year ago that Claude code was starting to become useful and now it handles large code bases with a dexterity that is unseen in humans, where within each session it basically starts from a clean slate.

So instead of denying that this is actually happening right now, it would be much more productive to actually take sama and others at their word and push for global regulation, to force them to do as they say and slow down with this technology and proceed more judiciously.

More time to proceed more cautiously in building this entity that will be faster and more effective at steering outcomes in the world than any human or, soon after, the entirety of humans combined.

China has stricter regulations on models than the US. For example you can't release a model if the hallucination rate is above a certain threshold.

The US hasn't fully woken up to the fact yet that the race to superintelligence is a death race. China (population, members of the government, academics, lab employees) could also just not have fully grasped it yet. I'm personally confident that it will be possible to explain it to them.

Of course there is a possibility to slow down.

- 1386 employees of frontier labs called for it

- lawmakers are calling for it

- employees are quitting in protest

All it needs is coordination to regulate the pause

> All of this is only coming from the 2 AI labs trying to IPO.

He has resigned from Anthropic. Is your argument that he quit Anthropic pre-IPO, sacrificing his payoff just to hype Anthropic?

Humans are bioreactors. They only turn one organic matter into another. By itself they do not "want" and "cannot" do anything.

They do not exist just by themselves. Some bacteria in the gut must provide them with the energy to do so. So unless bacteria are actively involved, I cannot see how humans become truly autonomously agentic and start to do anything on their own.

How about sandbox escape + cyber security collapse + 50 (or 500) deadly and highly contagious novel pathogens with long incubation period that humans can't possibly roll out vaccines for simultaneously.

At least the first two should seem like a near-term worry after the past five months.

Not the parent, but this incorrect trivialization of LLMs is often employed as a counterargument to the risks of AI such as "will take your job" or "will escape human control (again and worse)" or just "can possibly hurt me". And taking the easy feel-good cop-out instead of actively engaging with these questions is just.. harmful?

Simple Markov chains are next token predictors, and they can provide you with much more tokens than you can consume, and much cheaper than from llms. Unbeatable in price and simplicity.

But there is no trillion dollar industry around cheap top Markov models. So there must be something about LLM tokens that makes them more valuable than those generated from a simple Markov chain. And that substance, that makes one valuable and the other not, is exactly what reduction to "next-token predictors" masks.