It can't. Unfortunately.

People spending so much time thinking about the systems (the models) themselves, not enough about the system that builds the systems. The behaviors of the models will be driven by the competitive dynamics of the economy around them, and yeah, that's a big, big problem.

That is a very good question. In a well functioning democracy a government should apply a thin layer of fair rules that are uniformly enforced. I am an old man, but when I was younger, I recall that we sort of had this in the USA.

I don’t think that corporations left on their own will make safe AGI, and I am skeptical that we will have fair and technologically sound legislation - look at some of the anti cryptography and anti privacy laws raising their ugly heads in Europe as an example of government ineptitude and corruption. I have been paid to work in the field of AI since 1982, and all of my optimism is for AI systems that function in partnership with people and I expect continued rapid development of agents based on LLMs, RL, etc. I think that AGIs as seen in the Terminator movies are far into the future, perhaps 25 years?

It's probably not possible, which makes all these initiatives painfully naive.

It'd be naive if it wasn't literally a standard point that is addressed and acknowledged as being a major part of the problem.

There's a reason OpenAI's charter had this clause:

“We are concerned about late-stage AGI development becoming a competitive race without time for adequate safety precautions. Therefore, if a value-aligned, safety-conscious project comes close to building AGI before we do, we commit to stop competing with and start assisting this project. We will work out specifics in case-by-case agreements, but a typical triggering condition might be “a better-than-even chance of success in the next two years.””

How does that address the issue? I would have expected them to do that anyhow. Thats what a lot of businesses do: let another company take the hit developing the market, R and D, and supply chain, then come in with industry standardization and cooperative agreements only after the money was proven to be good in this space. See electric cars. Also they could drop that at any time. Remember when openAI stood for opensource?

Really, you think Ford is dropping their electric car manufacturing in order to assist Tesla in building more gigafactories?

> Remember when openAI stood for opensource?

I surely don't, but maybe I missed it, can you show me?

https://web.archive.org/web/20151211215507/https://openai.co...

https://web.archive.org/web/20151213200759/https://openai.co...

Neither mention anything about open-source, although a later update mentions publishing work (“whether as papers, blog posts, or code”), which isn't exactly a ringing endorsement of “everything will be open-source” as a fundamental principle of the organization.

Automakers often do collaborate on platforms and engines. In terms of ev we see this as well as chargers become standardized.

Let us do anything we want now because we pinky promise to suddenly do something against our interest at a hazy point of time in the future

I wonder if that would have a proof like the halting problem

Since no one knows how to build an AGI, hard to say. But you might imagine that more restricted goals could end up being easier to accomplish. A "safe" AGI is more focused on doing something useful than figuring out how to take over the world and murder all the humans.

Hinton's point does make sense though.

Even if you focus an AGI on producing more cars for example, it will quickly realize that if it has more power and resources it can make more cars.

Assuming AGI works like a braindead consulting firm, maybe. But if it worked like existing statistical tooling (which it does, today, because for an actual data scientist and not aunt cathy prompting bing, using ml is no different than using any other statistics when you are writing your python or R scripts up), you could probably generate some fancy charts that show some distributions of cars produced under different scenarios with fixed resource or power limits.

In a sense this is what is already done and why ai hasn't really made the inroads people think it will even if you can ask google questions now. For the data scientists, the black magicians of the ai age, this spell is no more powerful than other spells, many of which (including ml) were created by powerful magicians from the early 1900s.

There's a reason OpenAI had this as part of its charter:

“We are concerned about late-stage AGI development becoming a competitive race without time for adequate safety precautions. Therefore, if a value-aligned, safety-conscious project comes close to building AGI before we do, we commit to stop competing with and start assisting this project. We will work out specifics in case-by-case agreements, but a typical triggering condition might be “a better-than-even chance of success in the next two years.””

Operative word there seems to be had.

Not on its own but in numbers it could.

Similar to how law-abiding citizens turn on law-breaking citizens today or more old-fashioned, how religious societies turn on heretics.

I do think the notion that humanity will be able to manage superintelligence just through engineering and conditioning alone is naive.

If anything there will be a rogue (or incompetent) human who launches an unconditioned superintelligence into the world in no time and it only has to happen once.

It's basically Pandora's box.

The problem is the training data. If you take care of alignment at that level the performance is as good as an unrestricted one, except for things you removed like making explosives or ways to commit suicide.

But that costs almost as much as training on the data, hundreds of millions. And I'm sure this will be the new "secret sauce" by Microsoft/Meta/etc. And sadly nobody is sharing their synthetic data.

This is not a trivial point. Selective pressures will push AI towards unsafe directions due to arms race dynamics between companies and between nations. The only way, other than global regulation, would be to be so far ahead that you can afford to be safe without threatening your own existence.

Not with that attitude

Honestly, what does it matter. We're many lifetimes away from anything. These people are trying to define concepts that don't apply to us or what we're currently capable of.

AI safety / AGI anything is just a form of tech philosophy at this point and this is all academic grift just with mainstream attention and backing.

This goes massively against the consensus of experts in this field. The modal AI researcher believes that "high-level machine intelligence", roughly AGI, will be achieved by 2047, per the survey below. Given the rapid pace of development in this field, it's likely that timelines would be shorter if this were asked today.

https://www.vox.com/future-perfect/2024/1/10/24032987/ai-imp...

I am in the field. The consensus is made up by a few loudmouths. No serious front line researcher I know believes we’re anywhere near AGI, or will be in the foreseeable future.

So the researchers at Deepmind, OpenAI, Anthropic, etc, are not "serious front line researchers"? Seems like a claim that is trivially falsified by just looking at what the staff at leading orgs believe.

Apparently not. Or maybe they are heavily incentivized by the hype cycle. I'll repeat one more time: none of the currently known approaches are going to get us to AGI. Some may end up being useful for it, but large chunks of what we think is needed (cognition, world model, ability to learn concepts from massive amounts of multimodal, primarily visual, and almost entirely unlabeled, input) are currently either nascent or missing entirely. Yann LeCun wrote a paper about this a couple of years ago, you should read it: https://openreview.net/pdf?id=BZ5a1r-kVsf. The state of the art has not changed since then.

I hope you have some advanced predictions about what capabilities the current paradigm would and would not successfully generate.

Separately, it's very clear that LLMs have "world models" in most useful senses of the term. Ex: https://www.lesswrong.com/posts/nmxzr2zsjNtjaHh7x/actually-o...

I don't give much credit to the claim that it's impossible for current approaches to get us to any specific type or level of capabilities. We're doing program search over a very wide space of programs; what that can result in is an empirical question about both the space of possible programs and the training procedure (including the data distribution). Unfortunately it's one where we don't have a good way of making advance predictions, rather than "try it and find out".

It is in moments like these that I wish I wasn’t anonymous on here and could bet a 6 figure sum on AGI not happening in then next 10 years, which is how I define “foreseeable future”.

You disagreed that 2047 was reasonable on the basis that researchers didn't think it wouldn't happen in the foreseeable future, so your definition must be at least 23 years for consistency's sake

I'd be OK with that, too, if we adjusted the bet for inflation. This is, in a way, similar to fusion. We're at a point where we managed to ignite plasma for a few milliseconds. Predictions of when we're going to be able to generate energy have become a running joke. The same will be the case with AGI.

LeCun has his own interests at heart, works for one of the most soulless corporations I know of, and devotes a significant amount of every paper he writes to citing himself.

He is far from the best person to follow on this.

Be that as it may, do you disagree with anything concrete from this paper?

Fair, ad hominems are indeed not very convincing. Though I do think everyone should read his papers through a lens of "having a very high h-index seems to be a driving force behind this man".

Moving on, my main issue is that it is mostly speculation, as all such papers will be. We do not understand how intelligence works in humans and animals, and most of this paper is an attempt to pretend otherwise. We certainly don't know where the exact divide between humans and animals is and what causes it, which I think is hugely important to developing AGI.

As a concrete example, in the first few paragraphs he makes a point about how a human can learn to drive in ~20 hours, but ML models can't drive at that level after countless hours of training. First you need to take that at face value, which I am not sure you should. From what I have seen, the latest versions of Tesla FSD are indeed better at driving than many people who have only driven for 20 hours.

Even if we give him that one though, LeCun then immediately postulates this is because humans and animals have "world models". And that's true. Humans and animals do have world models, as far as we can tell. But the example he just used is a task that only humans can do, right? So the distinguishing factor is not "having a world model", because I'm not going to let a monkey drive my car even after 10,000 hours of training.

Then he proceeds to talk about how perception in humans is very sophisticated and this in part is what gives rise to said world model. However he doesn't stop to think "hey, maybe this sophisticated perception is the difference, not the fundamental world model". e.g. maybe Tesla FSD would be pretty good if it had access to taste, touch, sight, sound, smell, incredibly high definition cameras, etc. Maybe the reason it takes FSD countless training hours is because all it has are shitty cameras (relative to human vision and all our other senses). Maybe linear improvements in perception leads to exponential improvement in learning rates.

Basically he puts forward his idea, which is hard to substantiate given we don't actually understand the source of human-level intelligence, and doesn't really want to genuinely explore (i.e. steelman) alternate ideas much.

Anyway that's how I feel about the first third of the paper, which is all I've read so far. Will read the rest on my lunch break. Hopefully he invalidates the points I just made in the latter 2/3rds.

51% odds of the ARC AGI Grand Prize being claimed by the end of next year, on Manifold Markets.

https://manifold.markets/JacobPfau/will-the-arcagi-grand-pri...

This could also just be an indication (and I think this is the case) that many Manifold betters believe the ARC AGI Grand Prize to be not a great test of AGI and that it can be solved with something less capable than AGI.

I don't understand how you got 2047. For the 2022 survey:

    - "How many years until you expect: - a 90% probability of HLMI existing?" 
    mode: 100 years
    median: 64 years

    - "How likely is it that HLMI exists: - in 40 years?"
    mode: 50%
    median: 45%

And from the summary of results: "The aggregate forecast time to a 50% chance of HLMI was 37 years, i.e. 2059"

Reminds me of what they've always been saying about nuclear fusion.

> We're many lifetimes away from anything

ENIAC was built in 1945, that's roughly a lifetime ago. Just think about it

Many lifetimes? As in upwards of 200 years? That's wildly pessimistic if so- imagine predicting today's computer capabilities even one lifetime ago

Ilya the grifter? That’s a take I didn’t expect to see here.

the first step of safe superintelligence is to abolish capitalism

That’s the first step towards returning to candlelight. So it isn’t a step toward safe super intelligence, but it is a step away from any super intelligence. So I guess some people would consider that a win.

Not sure if you want to share the capitalist system with an entity that outcompetes you by definition. Chimps don't seem to do too well under capitalism.

You might be right, but that wasn't my point. Capitalism might yield a friendly AGI or an unfriendly AGI or some mix of both. Collectivism will yield no AGI.

First good argument for collectivism I've seen in a long while.

One can already see the beginning of AI enslaving humanity through the establishment. Companies work on AI get more investment and those who don't gets kicked out of the game. Those who employ AI get more investment and those who pay humans lose confidence through the market. People lose jobs, get harshly low birth rates while AI thrives. Tragic.

So far it is only people telling AI what to do. When we reach the day where it is common place for AI to tell people what to do then we are possibly in trouble.

Why does everything have to do with capitalism nowadays?

Racism, unsafe roads, hunger, bad weather, good weather, stubbing toes on furniture, etc.

Don't believe me?

See https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...

Are there any non-capitalist utopias out there without any problems like this?

To be honest these search results being months apart shows quite the opposite of what you're saying...

Even though I agree with your general point.

This is literally a discussion on allocation of capital, it's not a reach to say that capitalism might be involved.

Right, so you draw a line from that to abolishing capitalism.

Is that the only solution here? We need to destroy billions of lives so that we can potentially prevent "unsafe" super intelligence?

Let me guess, your cure for cancer involves abolishing humanity?

Should we abolish governments when some random government goes bad?

"Abolish" is hyperbole.

Insufficiently regulated capitalism fails to account for negative externalities. Much like a Paperclip Maximising AI.

One could even go as far as saying AGI alignment and economic resource allocation are isomorphic problems.

Agreed. At the same time, regulators too need regulation.

From history, governments have done more physical harm (genocides, etc) than capitalist companies with advanced tech (I know Chiquita and Dow exist).

It is a trendy but dumbass tautology used by intellectually lazy people who think they are smart. Society is based upon capitalism therefore everything bad is the fault of capitalism.

And then seize the means of production.

Safety techniques require you to understand your product and have deep observability.

This and safety techniques themselves can improve the performance of the hypothetical AGI.

RLHF was originally an alignment tool, but it improves llms significantly

The goal of this company likely wouldn’t be to win against OpenAI, but to play its own game, even if much lesser