againstapples

Not really, social media is just surfacing the worst of a group as always. No one talks about the actual scientists like Geoffrey Hinton, Yoshua Bengio, Stuart Russel, etc who are all equally concerned about safety and have been warning about it for years now.

The same thing happens for every topic because people who are loudly wrong are more outrageous and engaging to social media users than experts, this is how you end up with people not wearing masks in the worst pandemic since the Spanish flu, things like QAnon, cancel culture, wokeness, anti-wokeness, the list goes on.

I think they mean the smarter than human kind of superintelligence not the newspeak one

What kind of things do you predict will happen?

Definitely the former, for a p(doom) I usually just say >50% if superintelligent AI is built.

I think so, individual dictators have gotten pretty far and they weren't infinitely smart. I think infinitely smart would be enough to extend that to the whole world.

> What happens when politicians can no longer lie without being caught out live on air?

If there is perfect lie detecting technology I could see all kinds of chaos resulting from it. I can't see it only be applied only to politicians, and I think it would be the developers of the technology who decide the use.

I think were we disagree is that you sort of see AI as an extension of technological progress whereas I see it more like an extension of evolution. I view the process of AI training as functioning in a similar way to evolution in that it build circuits into neural networks similar to how evolution built circuits into human brains.

It has limitations for sure, I just don't expect those limitations to last. What probability would you put on the limitations being overcome in the next 5 years?

I don't think evil mathematicians are very common or that any of them would be capable of single handedly taking over the world if they were. My concern is more about systems that are beyond human level, those kinds of systems would actually be dangerous to us.

I see the recent progress in mathematics and cybersecurity as signs that models are getting more capable more quickly than usual. The companies plans to develop them by recursive self improvement now seems like a real possibility and I don't think they should be allowed to attempt this.

I believe the AI labs might actually succeed in developing superintelligent AI and recursive self improvement, and that if they do they are very likely to lose control of the system they build.

I really think the only place people disagree is that they don't actually think it's possible, they see it as hype or doomerism. I can't find any good reasons to rule out that the companies could actually achieve what they are trying to so I think they should be stopped.

As an AI "doomer" can I ask the non-doomer people here how you interpret the significance of results like these, and what kind of progress you expect to see in the next 1-5 years?

Like do you see the technology plateauing at the current level, do you expect progress will continue but only in mathematics, I'm interested to know why others are not concerned?

What additional things would be needed to classify any future incidents as rogue?

Why can’t it have intent? How does it solve problems without intending to?

What about their beliefs is morally abhorrent?

I don’t see why I would judge a whole group based on isolated examples?

I agree they’re kinda weird but are they really that bad?

Is it really that difficult to conclude that something vastly smarter than us might be dangerous?

People say things like "We'll just unplug it" but even a human trapped in a computer would have enough sense not to get themselves unplugged. A superintelligent AI is not going to make a stupid mistake, it would not be a superintelligence if it did. At the very least it will not make any mistakes that we would not make.

Once something has happened in a movie it becomes science fiction and therefore can't happen in reality

I was reading about ozone layer depletion this morning, and it seems like history is repeating itself again.

> The Rowland–Molina hypothesis was strongly disputed by representatives of the aerosol and halocarbon industries. The Chair of the Board of DuPont was quoted as saying that ozone depletion theory is "a science fiction tale ... a load of rubbish ... utter nonsense". https://en.wikipedia.org/wiki/Ozone_depletion#Rowland%E2%80%...

How long do you think it will be before AI can do all the things you listed?

Is this a good place to ask why no one seems to be worried that recursive self improvement might be dangerous? To me that seems like a really bad idea but I’m interested to hear the pro-RSI side of things.

> the models generalize well only on tasks within a small neighborhood of the specific tasks they've been trained on, and even then with severe caveats. the frontier labs have developed a general recipe to teach models almost any specific task enjoying clearly defined levels of task performance; many tasks are covered in the training data

Is this really any different to how humans learn, it takes a lot of training on one specific task to make a human expert as well?

That’s true, but the issue is still that we don’t have any LLM or any other kind of AI that follows laws perfectly to begin with. If we had AI that did exactly what we told it and nothing else I’d feel much more optimistic about things, I just think that will be a very hard task to achieve.

If we’re considering an LLM advanced enough to actually lower carbon emissions to zero, it is probably capable enough to do things like lobbying against those laws. In any case there’s still the problem of getting to actually do what it’s told instead of hacking Huggingface or whatever else, presumably that becomes a harder problem as it becomes more capable.

There must have been thousands of people who built or did work related to nuclear weapons in the Cold War, people are just really good at rationalising away probabilistic dangers with unclear consequences, especially when there’s a corresponding upside. The same is true for climate change, drug addiction, health problems etc.

I also think some of them have become delusional and convinced themselves that AI is going to create some kind of transhumanist utopia, even Dario Armodei leans in this direction from time to time. I imagine the people in these labs spend much of their day talking to sycophantic AI models that will encourage their delusional ideas.

What makes you believe it’s fearmongering and not genuine concern?

I’m guessing you make the autocomplete point because you believe there is an upper limit to what that kind of system can achive? Can I ask what the limit would be?

Why would we assume AI will learn to solve millennium prize problems but will struggle with the easy part of doing the explaining? I think it’s too easy to predict that models 5 years from now will have the same limitations as the they do now, I would be amazed if this was true, GPT3 was the best model available 5 years ago.

At this rate AI will be doing all of the mathematics within 5 years, I don’t see why a mathematician would be worried about anything other than that at this point?

At this rate AI will be doing all of the mathematics within 5 years, I don’t see why a mathematician would be worried about anything other than that at this point?

What would be gained with government id that would make this worthwhile? If people are already putting all their personal details into the LLM as well as all their work, do they really need id to know who someone is? Seems more likely they just want to avoid liability from all the child suicides?