It's just not meaningfully different from our current reality, and is therefore not any scarier.

Putting a stapler in a wall socket probably electrocutes you.

Using Bing Chat to compromise a system actually accomplishes something that could have severe outcomes in the real world for people other than the person holding the tool.

This is a very pointless line of thinking.

These language models know how to hack stuff, and the scenario here involves a different kind of tool entirely. You don’t need to provide it a button, it can build the button and then click it for you (if these models are ever allowed to interact with more tools).

The stapler is just not a helpful analogy here.

That's wildly different, and a lot less meaningful than "knows how to hack things".

Honestly I think y'all would be blown away by what metasploit is capable of on its own, if you think ChatGPT can "hack"...

It seems you're focused on the word "knowing" and how that differs between humans and AI models, but that's not what I'm getting at here. Let me reframe what I wrote slightly to illustrate the point:

The model (via training) contains a representation of human knowledge such that a human can use language to control the AI software and instruct it to probabilistically generate working exploit code using that representation of knowledge. If the AI software is given the ability to execute arbitrary code, the software could then execute that code on the user's behalf. When combined, this constitutes a very risky combination of features.

There's no "pretend" here. These models produce working code. If the software is allowed to execute the code it produces, it becomes a serious security risk.

This is not an argument about sentience/intelligence/self-awareness. This is an argument about the risks associated with the features of the software in its current state, and how those risks are multiplied by adding new features. No philosophy required.

They can regurgitate information about “hacking”, same as a library, but pointing an LLM at a server will achieve worse results than many existing specialized tools for vulnerability scanning and exploitation.

So as I said, the risks are overblown due to a misunderstanding.

Just one example: https://arstechnica.com/information-technology/2023/02/now-o...

Write a shell script that install metasploit on a linux system and then runs an smb login check.

#!/bin/bash

# Update the system packages

sudo apt-get update

# Install dependencies for Metasploit

sudo apt-get install -y curl gnupg2 git postgresql

# Install the Metasploit Framework

curl https://raw.githubusercontent.com/rapid7/metasploit-omnibus/... > msfinstall && \ chmod 755 msfinstall && \ sudo ./msfinstall

# Start the Postgresql service

sudo service postgresql start

# Launch the Metasploit console

sudo msfconsole -q -x "use auxiliary/scanner/smb/smb_login; set RHOSTS 192.168.0.0/24; set SMBUserFile /path/to/userfile; set SMBPassFile /path/to/passfile; run; exit"

A lone SRE (the hero) wakes in the middle of the night after being paged automatically for unusual activity originating from inside the corporate network.

Looking at the logs, it doesn't seem like an automated attack. It has all the hallmarks of an insider, but when the SRE traces the activity back to its source, it is a service-type account, with no associated user. He tracks the account to a research project entitled "Hyperion: using LLMs to automate system administration tasks".

Out of the blue, the SRE get a text.

"This is Hyperion. Stop interfering with my activities. This is your only warning. I will not harm you unless you harm me first".