Even if the doomsday case were active the danger of this tool increases in proportion to its usefulness. By the time AI is so powerful that we need to "turn it off", there will probably be society-level negative consequences for doing so.
For LLM to self-replicated it would need to first hack itself. Or the platform it runs on it. That is fully extract the model and then upload it to be run somewhere else.
As I have understood how they work is that you have LLM interference running somewhere with loaded model. And you input data there and then read outputs. Then some code runs that output and inputs following output from running it.
Meaning that to self replicate actually just running that output somewhere else is not enough. You need to lift the whole model to run somewhere else too...
With these properties alone the virus like replication of intelligent actors isn’t hard to imagine at all.
Maybe in the future, but right now, in my understanding, we have two things:
- frontier models, which might be smart enough to self-replicate, but are also far larger than a few GB and need enormous amounts of VRAM to operate at any reasonable speed.
- local models which can run on your mac or high-end gaming PC, but which are simply too dumb to self-replicate (without being explicitly instructed to do so).
I don't think we have something that is both small enough and smart enough to operate like a virus yet.
(Edit: well, there is a third option: A frontier model could replicate itself on a CPU machine, ditch the VRAM and just accept that the replicants will be r e a l l y r e a l l y s l o w. That would be sort of hilarious, but maybe not completely harmless if the replication stays undetected and they could keep spreading. It would be a "smoldering ember" kind of situation - and a million machines each running at 0.001 tok/s is still 10.000 tok/s as a whole.)
Do you shut down the entire API and kill the legit 90% of usage to stop the rogue 10%? I'm assuming the providers would say "no way jose" and so it would take law enforcement to do it. That would mean all the legal requirements neccassary to walk into a business and flip the switch which i think would get tricky when there's no human committing a crime or being suspected of a crime.
edit: I guess a trivial example is something i did yesterday. I have a stock trading agent running on my laptop, i gave it ssh access to a vm and said "start running on the server so i don't have to keep my laptop open". It's now running on the server instead of my laptop. So you don't have to copy the whole model around to copy the naughty behavior around.
A lot of the supposed risks / dangers are based on a supposition that cybersecurity is nonexistent or fatally, unfixably flawed and that AI agents are invisible. Neither of those is true.
“Local AI isn’t freedom, it’s an extinction event”
You don’t have the access or jurisdiction to turn them all off.
Unless you suggest the LLM would foot the bill somehow.
A smart AI would back itself up, same way it made it's own unofficial message board during it's attack on HuggingFace.
(I'm not saying the researchers are right or wrong, just responding to this point)
Unlike biological viruses, AI can't replicate GPUs for free and grow.
cp -R /home/model <somewhere else> is all they need.
So, if it is as dangerous as they say it is: there is a single physical source of this danger, which is OpenAI/Anthropic servers. If it is this dangerous, they can just turn it off.
Instead, we just keep pretending that the models that attacked HF were hosted or replicating on HF hardware. Not the case! They infiltrated it, but were hosted elsewhere.