I think in this case it's a false premise though. There are plenty of real practitioners like Joshua Bengio and Geoffrey Hinton who have sounded the alarm on these topics, and no one can call them inexperienced with real systems.
It's unfalsifiable, too.
The idea that people like Geof Hinton are wrong because they don't understand SRE is pretty ridiculous. SRE skills are relatively straightforward vs AI theory.
Now consider that a computer program can replicate and change itself orders of magnitude faster than a human generation.
I've dealt with spaghetti systems that need such maintenance and I'm unconvinced that poor devops is a major barrier to the hypothetical of runaway AI. Devops people will probably be using AI to fix these issues soon enough if they aren't already.
However, in their defense, the real question is whether those types of problems are something this new class of AI could potentially excel at. I've been an AI skeptic for decades, having had real world industry experience trying to get it to solve problems, and this is the first technological advance that has actually surprised me with its capabilities.
AI Risk still doesn't rank above many current, pressing global problems for me personally, but it also doesn't seem ridiculous anymore.
A possible next stop is that the LLM becomes better than humans at designing new algorithms and hardware.
Ever worked somewhere where all the good devs have left and it's outsourced to a bunch of cheap labour? They find ways to keep it running for years. Maybe not doing the way you'd like (log rotates are stuck? Better turn off the logging system), but they make it work. Ops people like to overestimate their own importance like everyone else.
> So how, in some detail, do we get from big-ass mixture model of transformers to even self-operation? That’s got to be a pre-requisite for self-improvement right?
No? Model could propose a bunch of changes to its own code and let the human developers take care of deploying it. Of course the developers could theoretically review those changes carefully... but let's be real, they won't. In the worst case a human-level-intelligent AI could certainly email the outsourced ops staff and ask them to deploy tag x. More generally, it can just do whatever a new dev would do - read your wiki and figure out which commands to run from that. If there's an error, look up old email threads on it, or similar errors online, or try to match up a traceback against source code and look for common errors. We expect fresh grads to do this stuff all the time, it's really not that hard.
> Then there’s the reality of AI development thus far, which is that it comes in stops and starts. Clearly the past is no guarantee of the future, but it seems a damned sight better as a prior than the log scale and a ruler methodology employed by Yud or whoever.
WTF kind of logic is this? Daily weather goes up and down, so clearly extrapolating any kind of long-term trend in temperature is silly.
The basic premise is that Google pardon me Avogadro Corp is doing NLP email copy-editing, but it takes too many compute resources so in desperation the TL hand wave sets it loose to secure its own budget, and being good at writing convincing emails, well, one thing leads to another.
Now this is very hand-wavy and involves a lot of magical thinking and suspension of disbelief. But that’s ok, it’s a pulpy sci-fi novel for fun.
Now this fictional scenario is dramatically more specific, more detailed, and frankly more believable than anything anyone has linked on this thread.
And I have no issue with folks wanting to LARP this sort of thing in a capital-R Rationalist forum somewhere: it’s a premise that has fascinated sci-fi fans since Asimov wrote his 3 Laws and probably before that.
But when it starts hitting the mainstream press, K Street, freaking out everyday people, and being used to turbocharge fear-driven land-grabs around intellectual property precedent?
Yeah, it’s time to do better than “But what if I Have No Mouth and I Must Scream Was Real man?!?!”
My whole premise is that it doesn’t have better than human intelligence. Everyone glosses over how we go from ChatGPT 4 failing its own unit tests to geometrically self-improving hyper-intelligence.
The burden is not on skeptics of paperclip optimizers in the near term to prove it can’t happen. That which is not prohibited by the laws of physics is admitted by the laws of physics. Of course it’s possible in the abstract.
The burden on anyone advocating positions being taken seriously by the public, by government, by industry, etc. to show that these arguments are in the public welfare, and I contend that they are not.
Obviously if it can't improve its code then it can't improve its code. But you seem to think there's some barrier that means that even if it was a better-than-human programmer (implicitly for general-intelligence reasons), it wouldn't be able to geometrically self-improve. And it's not at all obvious (at least to me) what that barrier is. We can already throw ChatGPT or equivalent at a codebase, ask it to suggest improvements to the code, and it does; the hit rate isn't great, but it's already good enough that people are incorporating it into their workflows.
As far as I can see, that's enough. Yes, people who are worried about this don't spend a lot of time going through the details, just as people who are worried about the military staging a coup don't spend a lot of time going into how they will get weapons or coordinate their movements or what have you. Those things are pretty complex, but they're ultimately routine; we are right to gloss over them, and if someone thinks that those things make a coup impossible, the burden would be on them to explain why (maybe there's some specific protocol that means they can't get access to weapons without authorisation from the legitimate authorities - but you'd need to explain what that protocol actually is and justify why you expect it to be wartertight).
Your argument seemed to be that babysitting the kind of complex code that we use for AI is not merely complex, but so complex that it requires some unique human ability that we would not expect a ChatGPT-like AI to possess (implicitly, that it requires something a lot more difficult than making improvements to code). And I just don't see that. Put it this way: we would never expect a human worker at these companies who was smart enough to make improvements to the core AI code to get stuck when it comes to actually running it. So why would it be an issue for the AI? I do think the burden is on you here given that the whole premise is that the AI is smart enough to improve its own code (you're welcome to argue that it's impossible for an AI to ever get smart enough to improve its own code - but in that case all this talk about operational complexity is beside the point).
P(next_bpe_token|previous_bpe_tokens,rlhf_orderings,internet)
to Cyberdyne Systems Model T-100. That’s a bit of a cheeky oversimplification of a modern decoder, but the LLaMA2 code is there for anyone to read, it’s kinda that. It’s not GPT-4, there’s clearly some scale or some special sauce or both on the way from here to there, but it’s damned close.
LLMs aren’t even a particularly great example of “adversarial” AI: AlphaZero is a way better example of AI kicking the shit out of humans in a zero-sum scenario. If I wanted to scare people I’d be talking about the DeepMind stuff, not the OpenAI stuff.
But the real crux of my argument is that bad-faith arguments have a tendency to escape the control of those making them, and if a short-term advantage in the copyright treatment of model weights resulting from the commons is achieved by Yud posting about GPT and nuclear weapons in Time Magazine and scaring the living hell out of the lay public about a capricious digital deity in the process, that’s unlikely to be a genie that goes back in the bottle.
So I’d really like it if someone injected some: here’s how this could actually happen into the dialog.
https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-no...
Then did exactly what the quote is talking about, assuming we achieve better than human intelligence. > But you seem to think there's some barrier that means that even if it was a better-than-human programmer (implicitly for general-intelligence reasons), it wouldn't be able to geometrically self-improve.
The question is how exactly do we go from a below human intelligence to an above human intelligence. Without a clear path to that reality, making trade offs in order to protect against low probability worse case scenarios which require it doesn’t look like a good deal.
The reality that right now human level intelligences have a hard time even keeping these below human intelligence systems operating seems like a useful checkpoint. Maybe we can hold off on doing distasteful things like consolidating control until they can at least wipe their own bottoms as it were.
Well, sure, because a) the article covers that b) their whole argument makes no sense if their position is that AI simply can't ever achieve better than human intelligence. If the AI is never intelligent enough to improve its own code then none of the operational complexity stuff matters!
> The question is how exactly do we go from a below human intelligence to an above human intelligence.
The same way we got to the current level of artificial intelligence; old-fashioned hard work by smart people. The point is that, if you accept that slightly-better-than-human AI would be geometrically self-improving, then by the time we have slightly-better-than-human AI it's too late to do anything.
It’s the same thing lately with AI… Sure, it has slightly more merit than cryptocurrency, but it doesn’t do what people think it actually does — certainly not as well either.
AI is not sentient yet people write full paragraphs to it as if it were? They even expect such in response. This is LUNACY! This is a mass delusion! Real life is not a Star Trek TNG episode with a talking computer!
It’s not an accident either. Nobody wants to look like a fool when there is a lot of money at stake. So the fly by night snake oil salespeople thrive… Some technology will be developed by overly sincere people along the way, but that was never the real goal.
Do you know what really smart people do? They don't do any of the things that AI Risk says will happen, they get jobs at Google (or insert other pointless white collar job on wall street) moving widgets by 1 pixel to drive an ad engagement metric by half of a percentage and get paid $500k+ a year in total compensation. Why? Because it is a relatively low effort way to fund a lifestyle that allows enjoyment of your intellect through experiencing the world in other dimensions besides the mundane of work.
The non-biological aspect is the only reason why AI Risk has any credence at all /in our current state/, because AI isn't distracted by a desire to travel the world and explore nature, or to read all the great works of literature and write their own analysis of it, etc. Biological beings, at least at this point in time and in the probable sense, have emotions and therefore have desires that range beyond whatever machinations you expect from AI Risk.
But something else to consider is that so far biological super intelligence doesn't seem to get all that much smarter than what humans are at generally. If somebody is guarded against you, then it's very difficult (impossible) to fool them even if you are way smarter than they are. This could indicate that maybe there's a cap to how super intelligent AI can get as well.
The main one is that I’ve never heard anyone describe in any detail how the technology will work. I’ve worked in big shops running big online ML systems and oh boy, do they need a lot of diaper-changing to even stay running. If all the ops folks went on vacation it’s a “hours vs days” not “weeks vs months” question of how soon it would fall over. Some log rotate thing gets stuck, and away we go with cascading failures.
So how, in some detail, do we get from big-ass mixture model of transformers to even self-operation? That’s got to be a pre-requisite for self-improvement right? inb4 “The risk is so great the details don’t matter because not impossible”, hmmm, no. Extreme tail risk isn’t interesting when addressing it comes at the cost of driving immediate, overwhelmingly likely risk of nightmare outcomes through the roof. Consolidating the alignment and steering of big models into the hands of a few CEOs, governments captured by them, or both is a clear and present danger of horrifying proportions. I take mitigate that over extreme tail risk sure as Tuesday and taxes.
Then there’s the reality of AI development thus far, which is that it comes in stops and starts. Clearly the past is no guarantee of the future, but it seems a damned sight better as a prior than the log scale and a ruler methodology employed by Yud or whoever.
Smart people talk like this is serious. There are even some elite practitioners who talk kinda like this (Hinton, Karpathy on Lex, some others). And I don’t want to be walking around with my head up my ass on it, so I’ll be grateful to anyone who wants to set me straight.
But if there’s an explanation somewhere, by a credible practitioner, of how this could actually happen in some actual technical detail, I haven’t found it.
I think that needs to be in the footnotes the next time the mainstream press starts publishing blog posts off LessWrong as consensus.