I just view the current gpt llm as a better search engine. It’s not self learning. It’s stateless when queried - so really the concern is we can create content on demand and create responses that intuitively make the llm very useful tool. Until it’s self learning I’m only concerned with how humans would use it just like any other tool in the hands of humans it can be both super useful and super destructive.

"just like any other tool in hands of humans" -- the scale of damage a tool can do matters, e.g. civilization cannot withstand nukes being sold at walmart. and, of course, self learning will be here this decade.

I agree llms are powerful but they are not explosive - conflating the two is IMO even more dangerous

they have the potential to take over the control systems of the explosive stuff. including the decision making processes of humans themselves.

to quote none other than sama, oct 2023: "i expect ai to be capable of superhuman persuasion well before it is superhuman at general intelligence, which may lead to some very strange outcomes" https://twitter.com/sama/status/1716972815960961174

take any narrow capability and 'superhuman <narrow capability>' will almost certainly be possible before 'superhuman general intelligence'.

The biggest, best transformer-based LLM you could imagine, that we haven't even made yet, will still fail the basic test "what is the weather tomorrow" since a LLM has no knowledge outside it's training set.

If they figure out how to give an LLM infinite/really long context length, it will basically become self-learning.

Which is more likely to happen first, "an LLM (with) infinite/really long context length," or an LLM-like system which can integrate new training data in real-time?

Am I even asking this question correctly?

LLMs can integrate new information but they only remember what you told them up to the context length. If they could remember longer then you would have agents with a lot of new knowledge.

I guess what I am getting at is: When I ask ChatGPT 4 Turbo to "search for the article by Eric Berger from Sep 10th 2023, and summarize it," it does, and responds including that new information because it is now in the context.

The next step would seem to be to integrate those search results into its training data - while somehow filtering bunk data which is the tough part - and when the next user asks that question it already knows it without searching.

Is this the same thing as saying "infinite context length?" I didn't think so. Is it?

What I mean is if context length wasn't a limit, you could have your own AI agent that remembers that article forever, along with everything else you've ever discussed with it.

If it generated text on its own, without human interaction, and developed its "personality", then you could say it's more like what we consider an intelligent being.

Context length is limited by the length of articles in the training data. You couldn’t use infinite context lengths unless the LLM had been trained on infinity long articles.

Context lengths aren’t arbitrary limits - they reflect the ability (and limits) of the model to actually reason about lengthy topics. If you want longer context lengths you’ll need more training data, larger models, and a lot more money.

Thank you.

So in my very uniformed mind.. "AGI," at the very least, would require real-time retraining... correct?

IMO yes, but more likely and entirely different yet to be discovered way of modeling intelligence

> any first AGI will only be able to run on large data centers

Until you task your newfangled AGI to optimize its own code so you can get your cloud server costs down.

Usually “mundane” vs “existential” harms/risks.

> ASI/AGI safety is only a concern right now if you believe in foom

False. There are plenty of fast takeoff scenarios that look real bad that aren’t “foom”. For example if you think AGI might be achievable within 5 year right now, and alignment is 10 years away, then you are very worried and want to slow things down a little.

No, then you start worrying when the technology is there. There is no point in worrying about it now when it is fiction. If AGI/ASI does not arise in some kind of uncontrollable "foom", you can always examine what you are dealing with right now and act accordingly. If AGI is achievable in a 5 years takeoff scenario then we have plenty of time and steps along the way fo study what we have and examine its danger. Right now we do not see AGI on the horizon and there is no significant danger from our models, so no reason to worry yet. If it happens, then long before it is "too late to stop it" we will see that GPT-6 or whatever is quite good and potentially dangepurs at certain tasks. Then you can go ahead and regulate. There is no point in doing it now.

On a sidenote, I don't really believe we can figure out anything in alignment anyways without having access to the models of corresponding capability, so waiting 5 years for alignment to catch up will so nothing.

If AGI is achievable in 5 years, then we do need to worry now, because we currently have no idea how to solve problems like prompt hijacking or how to durably convey our values to these systems.

> There is no point in worrying about it now when it is fiction.

I find this sort of epistemic nihilism completely baffling. Just because we have wide error bars, doesn’t mean we have to give up on trying to steer the future. If your 95% confidence window for “path A” includes ruin, and “path B” has lower EV but no ruin, why would you pick A?

The problem with alignment right now is we can’t even align GPT-3.5, let alone GPT-4. I believe this is a tech tree. New things building on old things at each generation. If we can’t keep up with the frontier, how will we align GPT-Omega? Concretely, things like Mechanistic Interpretability seem required. If we could pause progress for 5-10 years I feel optimistic that we could fully understand GPT-4 and how it actually renders concepts and decisions. (I don’t think we can pause progress, this is my case for why we should be at least a bit worried, and invest much more in interpretability research.)

It's not nihilism, I just expect absolutely nothing productive to come out of an attempt to regulate and control a technology that doesn't exist yet. Please show me a single result of alignment research that has real world applicability and is not just a variant of "train your model on aligned data so it is aligned". And the latter form of alignment success, RLHF and instruction finetuning, came after the base model capability was unlocked.

If you are doing any sort of challenging intellectual endeavour you need one of two things in order not to be inevitably wrong: you either use rigor as in math or the scientific method, or you do engineering with real things that you can test and play around with.

If neither of these two are giben then your field is likely to produce decades worth of junk results and predictions such as it happens in economics, sociology, psychology, and especially philosophy. You could fund a billion dollar 10 year program with economists in order to prevent the next big market crisis and they would not produce any valuable work.

I see AGI alignment research as being kind of in this position: Rigorous mathematical results are impossible or impractical (theoretical results about perfect actors maximizing utility functions or sth are unlikely to apply on the real world), and if you do not have AI models of that capability level available, you can also not engineer, test, and play around with your ideas. Hence I lots and lots of papers and theories will be produced and eventually the real world will show what was correct, but no valuable predictions will have been obtained.

> an attempt to regulate and control a technology that doesn't exist yet.

I’m not even proposing regulation. Just trying to understand the technology. You say it doesn’t exist yet, but many AI researchers (including Sutskever and Hinton, pioneers that know as much about the state of the art as anyone alive) think AGI could be achieved by scaling up the current architectures.

If there is a non-zero probability of AGI built atop transformers, we must invest more in understanding how transformers actually work. Even if it’s not that likely, it’s worth spending a lot more than we do now as a hedge. And I believe that if transformers do not end up being anything to do with AGI, it is likely that we will still carry over some expertise from catching up and understanding them.

For papers that are actually building our understanding here, see https://transformer-circuits.pub/. For example I think “Towards Monosemanticity”[1] was fairly widely considered to be a big step forward in the field.

There have also been meaningful results around steering models, which again relies upon being able to decode what features are actually encoding specific concepts.

I’m here for the idea that most papers are bunk, but isn’t that science in general? Really hard to pick good projects and researchers to invest in ahead of time, and so you kind of have to spray and pray at some level. Still worth it on net.

[1]: https://transformer-circuits.pub/2023/monosemantic-features/...

What about European regulations about genetic experimentation and GMO technology? The experiments in the broadest sense literally do not exist. They are not supposed to.

I mean, the GMO regulation papers are always referring to "nascent" technology. So existence is itself a dichotomous term for vague future capabilities.

So why can't people argue that alignment research is just a subset of the precautionary principle, which has already been applied to other such domains, at least in Europe? Precaution implies an absence of knowledge, but still having the agency for society and institutions to do something about it.

"Don't worry about country X's nuclear research program, they'll never make a bomb"

I do hope you're not in charge of any risk management.

If you're suggesting the emergence of an AI that can exponentially self-improve, coopt the resources to overwhelem human confrol, and magically build out the physical infrastructure sufficient to drive its globe-spanning humanity-squelching computational needs... that's a foom scenario.

If something that some people call AGI emerges in the next five years, (a) it's designation is going to be under tremendous dispute, (b) the infrastructure to enable a nightmare scenario won't be anywhere near reality.

Escaping that practical reality requires a complete ahistorical, ascientific faith in "yeah, but what if it's that capable?!?! The point is we can't know!!" that more strongly recalls history's secretive, sometimes dangerous, perennially incorrect apocolypse cults and alchemical societies than it does the diligent and reasoned scientists we might usually think of handing hundreds of billions of dollars.

Foom specifically refers to a very fast takeoff to ASI, usually hours/days or at the outside weeks. It’s a term of art, well-defined.

https://www.lesswrong.com/posts/LF3DDZ67knxuyadbm/contra-yud...

It does not refer to a fast takeoff where an AGI self-improves to ASI over multiple years.

It’s also not referring to the AGI threshold. If foom happens it’s probably unambiguous; there is now an entity that is far more intelligent than humans.

I think the foom scenarios are fairly unrealistic, for thermodynamic reasons. But I think it’s perfectly plausible that an ASI could persuade the company that built it to keep it secret while it acquired more resources and wealth over the course of years.

Your comment is a bit ambiguous, so you think that ASI/AGI safety will be important in the future as we get more compute and are less compute limited? Foom seems difficult to achieve with current LLMs, but I think there is always a possibility that we are just one or a few algorithmic breakthroughs away from radically better scaling or self-play like methods. In the worst case, this is what Q* is.

There is foom as in decently fast self improvement to super human level, but that is far from a dangerous foom scenario where that AI then escapes. You can just turn it off. It can not spread through the internet or something like that, that seems extremely unrealistic to me.

I'm not worried about foom at the moment because of the scale/power required.

The following is a bigger potential risk in my eyes. We keep gradually growing AI ability while rapidly growing compute capacity and miniaturization. Then in 10-20 years when we have data center capabilities in the size of a cellphone/laptop we have a massive AI algorithm improvement that brings ASI to something portable. How can you turn off something that can be ran on one of the hundreds of millions of computers already out there.

Q* reminds me of Q-learning and A* search, maybe it's somehow related to combination of such algorithms...

No. Foom is not needed. You could make a weak argument that AI training will be compute-limited but AI inference is of course much cheaper.

Yes. The AI safety that companies care about is their brand safety. They don’t want their products going Microsoft Tay on their customers.

We should in general terms not give AI - any AI, the ability to manifest itself in meat space. This alone almost certainly reduces the existential risks from AI.

Except many of the obvious use cases (including stuff like smart homes and military drones) are already being built.

You’d need very strong international laws to prevent this manifesting from happening.

also, manufacturing robots and self-driving cars.

More likely, you'd need the threat of imminent destruction, which, honestly, wouldn't be enough to prevent anything. We've seen how effective a death sentence is at prevention, which is to say: not very.

How can you stop that? An intelligent AI will send emails, create companies, hire people, and literally anything else you can do digitally in order to create means by which it can manifest itself into meatspace.

I don't mean to be facetious in any way, but how does AGI/ASI protect itself from humans simply pulling the plug?

Seems like it was pretty hard to pull the plug on sam? It will prevent you from pulling the plug by being smarter than you, spreading to more systems, making you dependent on it, persuasion,...

Neat idea, so it will conform like most humans in society and try to create more value than it destroys to avoid persecution and jail time? Sounds like our shared societal values will keep it in check then the same way they kept the openai board in check.

Why would it share our values? Human values are a result of our evolutionary history, which the AI will not share. We can't even formally describe them, so there's no hope of programming them into an AI. Of course, the AI will have to learn our values well enough to act in a convincingly friendly way while it's still weak, but knowing our values is not the same as sharing our values. Once it's gained enough power it can just kill us all. That's the most reliable way of ensuring we don't interfere with its goals.

GP was describing acquiring power, which is completely orthogonal to being aligned with our values. (Indeed, power-seeking is usually deemed to be a bad thing.)

There are certainly ecological and mimetic niches where pro-social behaviors will improve fitness. But it’s also certain that anti-social (defect/dominate/parasite) behaviors will improve fitness in many niches.

> It will prevent you

It will have no motivation to prevent you from pulling the plug. Human level intelligence is not the same as animal instinct.

If you had a system that was (a) oriented towards achieving some goal and (b) understood the world it was operating in, then allowing someone to "pull the plug" would interfere with it achieving the goal it was trying to optimize towards.

There has been some attempts at researching the question of how to design a intelligent system that is "corrigible" = willing to allow humans to change the goal it is set to optimize. This is unfortunately still an open question where no great solutions have been found that seem to be reliable when faced with a highly intelligent and capable AI system.

If you are interested in reading more, a few relevant search terms are "Off-switch game" and "corrigibility".

How do you “pull the plug” on a datacenter, or on all of the cloud providers if the ASI has processes running everywhere? Given that anyone with a credit card can already achieve robust multi-region deployments, it doesn’t seem hard for an ASI to make itself very hard to “turn off”.

Alternatively an ASI can ally with a group of humans to whom it can credibly promise wealth and power. If you think there is a baby machine god in your basement that is on your side, you’ll fight to protect it.

Airgap it. Give it no connections to the outside world, just a single controlled interface with a human operator. It's reduced to an advisor at this point, not an agent, but it removes most potential harm short of tricking its operators to plug it into the Internet.

In this scenario, pulling the plug is a matter of turning off power to the data centers it runs in - or simply disabling the one mode of external communication it has.

A sufficiently intelligent system that is un-aligned can likely subvert a human operator if it wants to.

But even if we ignore that, note that nobody is building their systems this way, and nor will they without extremely draconian laws requiring it. An airgapped system is substantially less valuable than one that is connected to the outside world.

I take the question “why can’t we turn it off” to refer to the actual real systems that we have built and will continue to build, not hypothetical systems we might build if we took safety risks very seriously.

I keep hearing this argument, and it is the worst one of all because it neglects human greed.

AI feeds on data. If it can make you a million dollars air gapped, you'll be able to make a billion with it plugged in the net with it manipulating data.

That may be, but as far as feasibility goes: I'd bet on solving a social problem over an amorphous technical problem (alignment).

Historically, the world has not been great at universally coordinating responses to social problems - especially when it only takes one actor to break the “truce.”

Isn't agreeing to only run an AGI with whatever theoretical alignment controls we come up with, also a social agreement? Seems we will have to figure that out one way or another.

Which is kind of the point... if we make AGI that it means we die. That is if the AI dormers predictions are correct.

The airgap option is “everybody agrees to leave lots of money on the table in the name of safety”. The alignment option is “some people invest lots of money to invent technology that everyone can use to safely make lots of money”.

If you solve alignment, the regulatory nudge required for folks to use it is probably not that big; let’s be really pessimistic and say it’s 10x the compute to run all the supervision. It’s probably mostly criminals that want the full-unaligned models, and so most entities can agree to the spend.

In the air gap case, I’m pretty confident that it is many orders of magnitude more profitable to run the “unsafe” / non-airgapped version. The utility of baking AI into everything is massive. Maybe in the best case you can have your airgapped ASI build provably-safe non-AGI tool AIs and capture much of that value, but I’m pretty skeptical it’s safe to transport anything that complex across the airgap.

So the alignment research path is more sustainable since it requires paying a much lower “safety premium”.

But I’m all for pursuing both, I don’t think they interfere with each other, quite the opposite; taking either one seriously reinforces the need for both of them.

Another advantage of the alignment path is we can meaningfully make progress now. It’s really hard to get people to agree ahead of time that they will run their AGI in an airgap.

A non foom scenario is a monopolistic and aggressive company maybe something like the 1990s Microsoft develops AGI and decides to monopolise the market the way they tried with Windows. Then later when they've got 90% market share and the AI has superhuman IQ, it turns. Thankfully the current Microsoft seems a bit less aggressive than the Gates version.