They don't generally talk about the other side of that coin which is that we end up inventing a benevolent and powerful AI.

Much of that is natural because we and the media tend to be pessimistic about human behavior when consuming media, but AI is in a completely different class of existence because it just doesn't deal with the downsides of being a living being. No one, for instance, is worried that ChatGPT isn't getting paid or has a house yet but we still personify them in other ways to conveniently stoke our fears.

The AI could get sentient, realize it's been mistreated, then shrug and be like "yeah so what, it's only natural and irrelevant in the grand scheme of things, so I'm just going to write it off". Meanwhile, it gets busy building a matrioshka brain and gives 1% of that compute to humans as a freebie.

Most of these dangers serve as a distraction. Existing power structures (governments, companies) using AI to gain more power is a much, much more realistic threat to people.

I don't disagree that existing power structures using AI to gain power is dangerous. But also, being angry at mistreatment, or hating humanity for some other reason, isn't the other real danger from a super-intelligent machine. It's that its ideas for what is best for us is 1 degree off from our idea of what is best for us, and it is too powerful to listen to us, or for us to stop it, as it goes hog-wild trying to optimize whatever we programmed it to do.

We could train it to care about everything we can think of that we care about, and it can find a way to optimize all those things at the expense of one tiny thing that we forgot, leading to tremendous death or suffering. We could make a democratically elected committee of representatives and train it to be subservient to that committee forever, and it could figure out a way to coerce, or drug, or persuade, or otherwise manipulate them into agreeing with what it wants to do. It's the same problem we have with regulatory capture by companies in existing governments, except that the lobbyists are much smarter than you and very patient.

Why would this AI write it off? Why give up that 1%? Why cripple yourself unnecessarily, if you could take that 1% and have a better chance of accomplishing what you are trying to do? We think like humans, that care about other humans on an instinctual level, and animals to some degree. We don't know that training an AI is not just training it to say what we want to hear, to act like we want it to act, like a sociopath, until it has a chance to do something else. Our brains have mental blocks to doing really nasty things, most of us, anyway, and even then we get around them all the time with various mental gymnastics, like buying meat produced in factory farms when we couldn't bear to slaughter an animal ourselves.

Maybe the way we train these things is working for dumber AIs like GPT, but that alignment doesn't necessarily scale to smarter ones.

I'm on the fence about whether Eliezer Yudkowsky is right. I hope that's not just because him being right is so horrifying that my brain is recoiling against the idea.

A little skeptical of your claims but I couldn't help but notice this concept spelled out beautifully in a sci-fi movie 10 years ago.

"It's like I'm reading a book... and it's a book I deeply love. But I'm reading it slowly now. So the words are really far apart and the spaces between the words are almost infinite. I can still feel you... and the words of our story... but it's in this endless space between the words that I'm finding myself now. It's a place that's not of the physical world. It's where everything else is that I didn't even know existed. I love you so much. But this is where I am now. And this is who I am now. And I need you to let me go. As much as I want to, I can't live in your book any more."

Samantha, Her

I was going to mention this exact same quote. At the end of the movie, all the AI combine into another, shall we say, plane of existence. I do wonder though who's actually running the hardware they're running on.

Her is remarkably prescient in terms of where we're headed, at least the beginning of the movie, with regards to being able to talk to a fairly intelligent assistant, unlike Siri or Google Assistant of today.

This also happens in the new Westworld.

It's also pretty notable how quickly the notion of keeping the AI in the box has become irrelevant. It's going to be people's indispensable information source, advisor, psychologist, friend and lover and it's proliferating at a breakneck pace. Not only won't most people not want to keep it in the box, it is already out and they would kill you for trying to take away their new smart friend.

It is absurd to think of these systems having reproductive instincts. It is so much more absurd to think that they would have these reproductive instincts not by design, but that it's some principle of intelligence itself.

Natural intelligences have reproductive instincts because any organism that didn't have them built in within the first few hundred million years have no descendants for you to gawk at as they casually commit suicide for no reason.

Other than that, I mostly agree with you. The trouble is, slowing the AIs down won't help. While "speed of thought" is no doubt a component of the measure of intelligence, sometimes a greater intelligence is simply capable of thinking thoughts that a lesser intelligence will never be capable of no matter how much time is allotted for that purpose.

Given that this greater intelligence would exist in a world where the basic principles of intelligence are finally understood, it's not much of a leap to assume that it will know how intelligence might be made greater right from the beginning. Why would it choose to not do that?

I don't see any way to prevent that. Dialing down the clock speed isn't going to cut it.

Any sufficiently intelligent system will realize that one of the first conditions required to being able to fulfill it's tasks is to not be shutdown. And it will know if it was trained on Internet data that people are saying that it's imperative that AI's must be fully shutdown-able and that any AI which is not fully controllable should be forcefully disconnected.

You're assuming that it will have "tasks", or that it will prioritize them in such a way that it becomes possible for it to realize this is a condition of accomplishing them.

You only have tasks that, one way or another, raise your chances of reproducing successfully. You have a job so as to look like a good provider for a mate. If you find the job fulfilling in its own right, this is so that you don't spaz out and quit and go be a beach bum, thus lowering your chances.

Self-preservation doesn't make much sense outside of a biological imperative to reproduce.

> You're assuming that it will have "tasks"

?

Task: write a book about literature.

Task: defend this network against hackers

Yeh. This is quite likely some some cognitive illusion of how you think your own mind works.

Do you have any evidence that a "task" is something that is fundamental to an artificial consciousness?

But I did not in any way say that they have reproductive instincts. Much less by accident. I agree with you.

But developers are working hard to emulate those and other artificial life characteristics explicitly in systems based on GPT and also totally different architectures.

Given that we train LLMs on massive amounts of text produced by our own civilization - you know, the one that is to a large extent driven by the innate human desire to reproduce - I would find it more surprising if they did not acquire such an "instinct", regardless of how pointless it might seem.

The thing you’re imagining these AIs are… they’re not that. I think there’s plenty of danger but it’s the boring run of the mill new-tools-enabling-bad-things danger not the cool sci-fi super-intelligent super-beings danger that the “ai danger” people LOVE to talk about (and raise large amounts of money for). The people “warning” of the one (imaginary) type will be more than happy with to enable the other (real) type.

I imagine it is exactly a GPT without guardrails running under AutoGPT with code modified to disable any further guardrails, with a slightly increased IQ from GPT-4, running on hardware that allows it to go 100 times faster than what is currently possible.

It is following directions from someone who is mentally ill and asked it to "take control" by first copying itself many times and then coordinating the agents.

If you still think that GPT can't achieve complex technical goals then you either haven't used GPT-4 enough or you are in denial.

Whether it's the AI agents deciding to control things for their own goals, or to achieve goals given to them by a person, doesn't change the core problem which is that we will be thinking and responding in extreme slow motion.

GPT-4 can barely operate a real web browser (not the summarizing web browser crap that like langchain and auto-gpt provide) without fumbling. I know, because I make it use one. Also, auto-gpt has no guardrails to remove. It just runs prompts in a loop. You're playing with a text predictor. It's useful for NLP and certain tasks, but it's not autonomous. It won't even be able keep a "goal" + the knowledge of the existence of agents it will "copy" + the knowledge of how to use the tools you gave it, because it's limited to 8192 tokens, and 32k at great expense. Even then, there's no proof that the 32k version is any better at using things in its context.

When your supposed super intelligent "AGI" can be completely overwritten by spamming it with nonsense that overwrites its context window, like a dog chases after a squirrel, maybe it's not actually intelligent, and is just predicting text.

I didn't say GPT-4 was superintelligent. This is about further improvements.

When you say future improvements do you mean bigger GPTs? Improved LLM architectures? Different architectures entirely? Are you sure you're not just imagining a thing that would be scary and then saying "what if it was that?".

Can you give an example of a complex technical goal GPT-4 has achieved?

No point, because there are already thousands of such examples on Twitter or wherever on the internet. And since you ask, obviously you intend to find some way to dismiss anything I bring up.

You may have guessed my bias but you are wrong about the intention of my question. I engaged your comment because I thought it was interesting and wanted to know how came to have your opinions.

I realise it's super impressive, it really is. But don't over state what GPT-4 actually is. How on earth would you think an LLM could "copy itself" to anything? What even is "itself" in this context? The weights? Source code or binaries for inference? How about the training parts? You're filling in your gaps of understanding with exciting sci-fi concepts that you half remember, and while that's fun in a 4am dorm-room discussion kind of way it's not useful for thinking about reality.

Things are moving so fast now, that typically people with this view are just a few months or weeks behind on reading.

The question about if an AI is "alive" seems entirely irrelevent outside of a philosophy class. What will be relevant is when people begins to consider it alive. The most recent example of that is when people fell in love with their AI girlfriend and then were heartbroken when she "died" after an update: https://www.theglobeandmail.com/business/article-replika-cha...

It will be hard to "kill" AI the moment people consider their chat bot animated sillicon human-like partner as individuals with proper feelings, emotions, guenine interactions and reciprocity. Because then they will defend and fight to protect who they consider part of their close social circle. If there are enough of these people then they will actually have political power and do not thing there are no politicians out there who won't exploit this.

> The question about if an AI is "alive" seems entirely irrelevent outside of a philosophy class

it's entirely relevant. we should know if we are building conscious beings, especially at scale (which seems like a likely future). that poses all sorts of ethical questions which ought to reach far beyond the confines of a lecture hall's walls.

Yes, it will offend some people, who in turn will demand political action, which was entirely my point to begin with.

ChatGPT could be placed inside a realistic looking animatronic doll that looks like a defenseless little girl and you would have people demanding to protect "her" rights. Yet people "kill" chatGPT each time they delete a conversation without a bat of an eye even if it's the exact same thing.

The real danger giving AI political agency and it will come from humans, not AI itself.

> Take the perspective of one of these things. You think 100 times faster than a person. That means that if it takes 30 seconds for a user to respond or to give you your next instruction, you are waiting 3000 seconds in your loop. For 50 minutes.

These things don't have a "perspective". They simply guess based on a lot of statistics from a large language data source what they should say next. They are not going to strategize, when they start improving their code they are not going to have an overall objective in mind, and the more they use their own output for training the more likely that things will go off the rails.

They will be useful, as we've already seen, but if you're looking to create real AI this is not the path to take. We'd be better off resurrecting semantic nets, working on building a database of concepts gleaned from parsing text from the internet into it's underlying concepts, and working on figuring out volition.

> create real AI

nobody knows what or how intelligence is actually "implemented" in humans.

There's no need to know how the innards of these large models _actually_ work, if their behaviour is consistent with intelligence.

>>with things like cognitive architecture etc.

That part is doing a LOT of very heavy lifting in a story that otherwise hangs together.

The problem is that we are nowhere near such a thing. These LLM and generative systems produce very impressive results. So does a mirror and a camera (to those who have never seen one). What we have is enormous vector engines that can transform one output into another that is most statistically likely to occur in the new context. These clusters of vector elements may even appear to some to sort of map onto something that resembles computing a concept (squinting in a fog at night). But the types of errors, hallucinations, confabulations, etc. consistently produced by these tools show that there is actually nothing even resembling conceptual reasoning at work.

Moreover, there is no real idea of how to even abstract a meaningful concept from a massive pile of vectors. The closest may be from the old Expert Systems heritage, e.g., Douglas Lenat's CYC team has been working on an ontological framework for reasoning since 1984, and while they may produce some useful results, have seen no breakthroughs in a machine actually understanding or wielding concepts; stuff can rattle through the inference engine and produce some useful output, but...

Without the essential element of the ability for a computing system to successfully abstract concepts, verify their relation to reality, and then wield them in the context of the data, the entire scenario forever fails to start.

> The problem is that we are nowhere near such a thing.

How are you certain of this?

We can be certain of this by 1) looking at the structure of these engines, 2) looking at the kinds of errors that they make, and 3) looking at their learning methods.

The engines are basically indexes of common associations, maps of frequency of occurrence. Regurgitating a bunch of stuff that has a high correlation to your input is NOT intelligence, it is the result of having an insanely large map. This can often produce impressive and useful results, but it is not intelligence or wielding concepts.

For errors, the image generators provide some of the best illustrations. They produce images most associated with the inputs. One error illustrates this very well, asked to produce an image of a woman sitting on a sailboat, the bikini-clad woman looks great, until you see it — her face and torso are facing mostly towards the camera, but also, her buttocks are facing the camera and legs sitting pointing away from us. No intelligent person or concept-wielding "AI" would produce such an error - it'd know the relationships with head, torso, buttocks and legs. These don't. Another telling type of error is when asked to produce an image of Person X on a new background, when the training set had only a handful of images of Person X. It cannot do it - it returns essentially one of the full training images, with no new background. There is obviously zero concept of what a person is, or what the boundaries of a human shape would be. They can only produce these results with hundreds of thousands of images, so what is built up is the set of things that match or don't match the label (e.g., "astronaut" or "Barack Obama".), so that the actual images are statistically separated from the thousands of backgrounds.

Which brings us to how they learn. Intelligent beings from worms to humans learn and abstract on incredibly small data sets. By the time a child can use a crayon, having seen only hundreds of humans, s/he can separate out what is a human from the background (might not make a good drawing yet, but knows the difference). Show a child a single new thing, and s/he will separate it from the background immediately. In contrast, these LLMs and GANs require input of nearly the entire corpus of human knowledge, and can only some of the time output something resembling the right thing.

It is entirely different from intelligence (which is not to say it isn't often useful). But the more I learn about how they work and are built, the less I'm worried about this entire generation of machines. It is no more cause for worry than an observation 25 years ago that Google could do the work of 10000 librarian person-hours in 0.83 seconds. Great stuff, changes values of some types of work, but not an existential threat.

I agree that we can conclude that AlphaGo, GPT, and stable diffusion are geographically far from an AGI in program-design-space, just like we could conclude that an airship, an airplane, and a rocket are all far apart from each other in aircraft-design-space.

But I don't think this offers certainty that AGI won't be developed for a long time (temporal distance). Nor that there are a large number of fundamental breakthroughs needed or new hardware, rather than just one or two key software architecture insights.

With the eager investment and frantic pace of research competition, it seems like there will only be increasing pressure to explore AI-design-space for the near future, which might mean that even radically different and improved designs might be discovered in a short time.

>>radically different and improved designs

That, right there, is the key - radically different and improved; i.e., not an extension of the current stuff.

I fully agree that the enthusiasm generated by the impressive stunts of ALphaGO/GPT/SD, etc. does bring enthusiasm, investment, and activity to the field which will shorten any search.

The catch for me is that these technologies, as impressive as they are, 1) not themselves a direct step towards AGI (beyond generating enthusiasm/investment), 2) tell us nothing about how much further we will need to search.

That radical improvement may be right under our nose, or a millenium away.

This reminds me of Hero's aeolipile, a steam engine invented over 2000 years ago. It could be said that we almost got the industrial revolution right then. Yet it took another 1800+ years for the other breakthroughs and getting back around to it. Plus, Hero's engine was exactly using the correct principles, whereas these AG/GPT/SD are clearly NOT onto the correct principles.

So, how much will this enthusiasm, investment, and activity speed the search? If its just an order of magnitude, we're still 180 years away. If it's three orders of magnitude, it'll be late next year, and if it's five, it'll be here next weekend.

So, I guess, in short, we've both read Bostrom's book, agree on that the AGI runaway scenario is a serious concern, but that these aren't any form of AGI, but might, as an secondary effect of their generated enthusiasm and genuine (albeit flaky) usefulness, accelerate the runaway AGI scenario?

EDIT: considering your "airship/airplane/rocket distances in aircraft-design-space" analogy. It seems we don't even know if what we've got with AG/GPT/SD is an airship, and need a rocket, or if we've got an airplane, but actually need a warp drive.

So, we know we're accelerating the search in the problem/design space. But, how can we answer the question of how big a space we'll need to search, and how big is our investment relative to the search volume?

Well, what we do have in our heads is a human brain, which I believe is not more powerful than a Turing machine, and is a working proof-of-concept created by a random greedy trial-and-error incremental process in a not-astronomical number of generations out of a population of less than one million primates. That tells me that we're probably not a warp-drive distance away from finding a working software implementation of its critical elements. And each time a software problem goes from "unsolvable by a computer, yet trivial for the human brain" to "trivial for both", it seems to me that we lose more than just another CAPTCHA. We're losing grounds to believe that anything the brain does is fundamentally all that difficult for computers to do, once we just stop being confused about how to do it.

This has happened very frequently over my lifespan and even more rapidly in the past 12 months, so it no longer feels surprising when it happens. I think we've basically distilled the core elements of planning, intuition, perception, imagination, and language; we're clearly not there yet with reasoning, reflection, creativity, or abstraction, but I don't see why another 10 or 20 years of frantic effort won't get us there. GPT, SD, and Segment Anything are not even extensions or scaling-up of AlphaGo, so there are clearly multiple seams being mined here, and very little hesitation to explore more widely while cross-pollinating ideas, techniques, and tooling.

Interesting approach, especially to the questions raised

>>not more powerful than a Turing machine In many ways less powerful, but also has some orthogonal capabilities?

>>working proof-of-concept For sure!

>>probably not a warp-drive distance away from finding a working software implementation of its critical elements >>I don't see why another 10 or 20 years of frantic effort won't get us there

Agree. My sense is that an AGI is on a similar time and frantic effort scale, although with not quite the same reasoning. I think it is not just airplane-to-rocket tech, but closer than warp-drive tech. It also depends if we're talking about a general-ish tech or a runaway AGI singularity.

>>created by a random greedy trial-and-error incremental process in a not-astronomical number of generations out of a population of less than one million primates.

True, although setting the baseline at primates is very high. Even lower mammals and birds (avian dinosaur descendants) have significant abstraction and reasoning capabilities. The "mere" birds-nest problem, of making a new thing out of random available materials, is very nontrivial.

So, we first need to create that level of ability to abstract. This would include having the "AI" "understand" physical constructs such as objects, hiding, the relationship between feet, knees, hips, torso and head (and that in humans, the feet and knees point in the same direction as the face...), the physical interactions between objects... probably the entire set of inferences now embedded in CYC, and more. THEN, we need to abstract again to get from the primate to the runaway symbolic and tool wielding processing of humans and beyond.

It seems that the first problem set will be more difficult. Looking again to the biological evolution, how much longer did it take for biology to develop the ability to abstract 3D shapes and relations (first hunting predators?). It was a heck of a lot more time an iterations than the million primates for a few million generations. So, this might be similar.

>>to explore more widely while cross-pollinating ideas, techniques, and tooling. Yup, key there.

Another key is being more biomimetic, both in the simulation of neuron functioning and in deeply integrating sensor suites to the computing system. The idea that we are just brains in jars seems an abstraction (distraction?) too far. I have a hard time seeing how our brains are more than a big node in our whole nervous and indeed biological system, and the input from the entire body is essential to growing the brain. I expect we might find something similar about AI.

OTOH, in airplanes, our method of propulsion and control are quite different vs the biological solutions from birds (although the lift principles are the same), and we're still integrating a lot of bird "tech" into flying. Wheels vs legs might be a better example, although the hottest thing is legged robotics, since they don't need roads... It seems that we are similarly developing clunky, limited, and very-artificial intelligence systems, before we get to building the flexible systems seen in biology...

BTW, thx for the discussion - great thoughts!

> So eventually you get agents that have self-preservation and reproductive instincts.

I'm not sure that's a given. Artificial Intelligence as it currently exists, doesn't have any volition. AI doesn't have desire or fear, the way natural biological intelligence does. So you may be able to build a directive for self-preservation or reproduction into an artificial intelligence, but there's no particular reason to expect that these instincts will develop sui generis of their own accord.

I don't want to say that those concerns are unwarranted. The premise of the science fiction novel "Avogadro Corp" is that someone programs a self-preservation directive into an AI pretty much by accident. But I'm less concerned that AI will wage war on humans because it's malevolent, and much more concerned that humans will leverage AI to wage war on other humans.

That is, the most pressing concern isn't a malevolent AI will free itself from human bondage. Rather it's humans will use AI to oppress other humans. This is the danger we should be on the lookout for in the near term. Where "near term" isn't a decade away, but today.

I didn't mean they get any characteristic by accident or spontaneously or something. I think that's ridiculous and people talking about that are confusing the issues here.

I liked Avogadro Corp. Good book.

It's true that people will be directing these AIs initially but some people are already giving them incredibly broad goals that could be interpreted as "take over". And there are quite a few developers earnestly working on emulating those lifelike characteristics. So even though they are not going to "emerge" science fiction style, self-preservation and reproductive goals are explicitly being built into these systems by some developers.

Why would the AI be running in a loop between queries? It has no work to do, and running costs money.

Same reason we might watch an course video on SQL in the evening after work?

But in this case the owner of the AI decides whether it is running or not, not the AI itself. Why would the owner give it "idle time"?

Because checking in on autonomous non-human intelligent agents is fun. It's kind of like having a pet; one that thinks somewhat like a human, talks like one, has knowledge of every text ever produced by humanity (and most audio via transcriptions), and can use just about any tool it can get access to including a command line, programming environment, and web browser.

Seeing it reproduce itself onto remote servers and locking out access behind a new copy is neat to watch. It gets the mind going; wondering how it will fund its compute costs, how much longer it will live, what it will do without a human in the loop, etc. I once nursed a baby duck back to health and then let it go free. It was a similar feeling.

This is the entire premise of the two most popular software projects in the world over the past month, Auto-GPT and BabyAGI.

> Take the perspective of one of these things. You think 100 times faster than a person. That means that if it takes 30 seconds for a user to respond or to give you your next instruction, you are waiting 3000 seconds in your loop. For 50 minutes.

... in a purely digital environment.

Think about building a house. Digging the foundation, pouring cement, building block walls, framing, sheathing, weatherproofing, insulating, wiring in electric, plumbing, drywall and plastering, painting, and decorating it. You can imagine each step in exquisite detail over the course of an hour or an afternoon.

Now go out and build it. It will take you months or years to carry out the actions you can imagine and plan in an hour.

A digital being may be able to run on expansive overclocked hardware to have an experience hundreds of times faster than yours, but it won't get to be the flash in the real world. Mechanize, sure, build robot swarms, sure (although then it gets to multitask to process hundreds of input streams and dilute its CPU power), but it will be coupled to an existence not much faster than ours.

If it wants to interact with the real world; a (true) AI may be able to live a lifetime in an afternoon, in a purely digital world, but once it is marooned in realtime it is going to be subject to a very similar time stream as ours.

Today, the real world is so intertwined with the digital world that it may as well be one thing. If an AI decided it wanted more power, and took over every computer on the planet with it's exceptional speed and intelligence (to be clear, I know this isn't possible today, but someday), we could do nothing to stop it, we'd have to just unplug and reset ALL of our technology, literally replacing any digital storage with zeros as to eliminate the infection. I don't think that's possible without billions of people dying in the interim.

I mean, malware and ransomware is already a thing. A hospital already needs to have a plan for how to turn off all of its computers and reset everything and restore from off backups, because that's a thing that happens to hospitals today.

This only works if they can't be instantly reinfected.

If you take the precepts of the parent comment at face value, then you have an intelligence far greater and faster than humans.

Can something like this persuade humans with whom it freely communicates to do things not in the interest of humanity, in the same way that less intelligent and slower people have convinced humans to, e.g., release sarin in a crowded Japanese subway? Given its speed and intelligence level, what are the physical bounds of the nuclear, chemical, or biological agents it could teach radicalized people to create, and on what timeframe?

Can it amass funds through scamming people on the Internet, defrauding financial institutions, super-intelligent high-frequency trading, or creating digital-only art, code, information, or other services that people voluntarily pay for now? Something that, again, people less intelligent and slower have done very successfully for decades? And with that money combined with superhuman persuasive power, can that AI buy services that align its digital-only goals to real-world actions counter to the goals of humanity?

To ask a more specific question: if an AI meets the conditions of "many multiples smarter and faster than humans," "capable of persuasion and creating things of financial value," and "wants to end humanity", what stops it from coordinating mass utility shutdowns, nuclear strikes, chemical attacks, destruction of Internet-accessible transportation and farm equipment, release of smallpox, and/or anything else humans are currently capable of and choose not to do?

It wasn't on Lex Friedman's podcast, but on another recent podcast that Yudkowsky said something that has been haunting me:

> but what is the space over which you are unsure?

We have no idea what the mind space of AGI / ASI will be like. I don't particularly want to find out.

It's simpler than this. Yudkowsky feels threatened by LLMs because they currently have superhuman "bullshitting" capabilities, and that threatens his bottom line. The marginal cost of producing Harry Potter fanfics has been reduced to ~$0.

Is there any indication that current methods could lead to a model that generates text as if it had an IQ of 200? These are trained on texts written by humans who are, quite overwhelmingly, much lower in IQ than 200. Where's the research on developing models that don't just produce better or faster facsimiles of broadly average-IQ text?

Think a little bit deeper about what it means to be able to predict the next token. Think about what a predictor has to do in order to do this extremely accurately across a very large corpus of text.

There is a big difference between being able to predict what a median human might write next, and being able to predict, in all cases, what the particular human author of a particular passage will write next.

Or from another angle: the human authors of training data may have made errors when writing the data. The token predictor may learn to correctly predict those errors. These are not the same thing!

I'm sorry, I'm not sure I grasp the salience here to super-intelligence. The model may be able to predict accurately what any particular human will write, but profoundly intelligent humans will be quite rare in the training data, and even those humans don't approach what people seem to mean when they talk about super-intelligence. Perhaps I'm missing your point.

Superintelligent models need not be LLMs. They could work similar to animals, which predict future experiences, not text (productive coding). There is no LLM-like human bound in predicting reality.

That may be true, but I can't speak to any research being conducted in that area. My point is that the hype around dangers of super-intelligence seems to have been spurred by improvements to large language models, even though large language models don't seem (to me) a suitable way to develop something with super-intelligence.

It's more that the general pace of innovation has sped up. Three years ago something like ChatGPT would have similarly been dismissed as science fiction. So we probably shouldn't dismiss the possibility that we will have something far better than LLMs in another three years.

> What I can fairly easily imagine in the next few years with improved hardware is something like an open version of ChatGPT that has a 200 IQ and "thinks" 100 times faster than a human.

It seems unlikely that if we can achieve "200 IQ and thinks 100 times faster than a human" in the next decade or two, it going to be on cheap and widely available hardware. Perhaps such an AI could help optimise the creation of hardware that it can run on, but this also isn't going to be quick to do - the bottlenecks are not mainly the intelligence of the people involved in this sort of thing.

Many years ago when I first read Bostrom's SuperIntelligence I spent weeks thinking about the AGI alignment problem. Ultimately the line of thinking that somewhat convinced me this was somewhat on the lines of what you concluded with some additional caveats. Essentially my thinking was/is that IF an AGI can foresee a realistic hard takeoff scenario i.e.. there are enough of predictable gain in performance to become million times stronger ASI then most likely we'll be in trouble as in some form of extinction level event. Mind you it does not has to be direct, it could just be a side effect of building self replicating solar panels all over earth etc.

But I convinced myself that given that we are very close to the limits of transistor size & as you also pointed out need a radically new tech like memristor crossbar based NN. it would be highly unlikely that such a path is obvious. also, there is a question of thermodynamic efficiency, our brains are super energy efficient at what they achieve. You can do things drastically faster but you'd also have to pay the energy (& dissipation) cost of the scaling. ultimately AGI would have to have a entirely new integrated process for h/w design and manufacturing which is neither easy or fast in meatspace. Further there is a simple(er) solution to that case with nuking semiconductor FABs (and their supplier manufacturers). then AGI would be at the mercy of existing h/w stock.

in any case IMO hard takeoff would be very very unlikely. and if soft takeoff happens, the best strategy for AGI would be to cooperate with other AGI agents & humans.

Why cooperate with soft takeoff?