News for hackers. Stuff that matters.
Re:The Last 24 Hours Are the Opening Scene in a Horror Movie by stephantul on Friday September 11, 2026 @12:27PM (#49661113)
One thing I definitely do not understand about this discourse is that the models that are good enough to self-replicate can’t survive on normal machines, e.g., the models can’t hide on some random server.

So, if it is as dangerous as they say it is: there is a single physical source of this danger, which is OpenAI/Anthropic servers. If it is this dangerous, they can just turn it off.

Instead, we just keep pretending that the models that attacked HF were hosted or replicating on HF hardware. Not the case! They infiltrated it, but were hosted elsewhere.

Re:The Last 24 Hours Are the Opening Scene in a Horror Movie by largbae on Friday September 11, 2026 @03:56PM (#49664457)
This assumes that everything an AI (or more likely an evil _user_ of AI) can do requires its active participation on D-day. Creating a virus that spreads like Covid but kills like Ebola would be complete as an AI use case long before the first person sneezed.

Even if the doomsday case were active the danger of this tool increases in proportion to its usefulness. By the time AI is so powerful that we need to "turn it off", there will probably be society-level negative consequences for doing so.

Re:The Last 24 Hours Are the Opening Scene in a Horror Movie by Ekaros on Friday September 11, 2026 @12:40PM (#49661353)
Either I have wrong mental model or then too many other people have wrong mental model.

For LLM to self-replicated it would need to first hack itself. Or the platform it runs on it. That is fully extract the model and then upload it to be run somewhere else.

As I have understood how they work is that you have LLM interference running somewhere with loaded model. And you input data there and then read outputs. Then some code runs that output and inputs following output from running it.

Meaning that to self replicate actually just running that output somewhere else is not enough. You need to lift the whole model to run somewhere else too...

Re:The Last 24 Hours Are the Opening Scene in a Horror Movie by xg15 on Friday September 11, 2026 @01:58PM (#49662548)
Exactly. But even if the model got access to its own weights, it would need to find a machine with a sufficiently capable GPU, transfer the weights and install itself there. Basically, the only machines powerful enough that AIs could self-replicate to would be other AI datacenters.
Re:The Last 24 Hours Are the Opening Scene in a Horror Movie by cdaringe on Friday September 11, 2026 @01:09PM (#49661765)
It’s not even hack though. I have my model and agent self modify its running parameters, thus self service, on commodity hardware. I have it run other models and other software. there is agent model autonomy here, and it’s not even complicated. A harness is kilobytes, a model is gigabytes, and networks are abundant. Moving the pieces is easy and cheap.

With these properties alone the virus like replication of intelligent actors isn’t hard to imagine at all.

Re:The Last 24 Hours Are the Opening Scene in a Horror Movie by xg15 on Friday September 11, 2026 @02:02PM (#49662625)
> a model is gigabytes

Maybe in the future, but right now, in my understanding, we have two things:

- frontier models, which might be smart enough to self-replicate, but are also far larger than a few GB and need enormous amounts of VRAM to operate at any reasonable speed.

- local models which can run on your mac or high-end gaming PC, but which are simply too dumb to self-replicate (without being explicitly instructed to do so).

I don't think we have something that is both small enough and smart enough to operate like a virus yet.

(Edit: well, there is a third option: A frontier model could replicate itself on a CPU machine, ditch the VRAM and just accept that the replicants will be r e a l l y r e a l l y s l o w. That would be sort of hilarious, but maybe not completely harmless if the replication stays undetected and they could keep spreading. It would be a "smoldering ember" kind of situation - and a million machines each running at 0.001 tok/s is still 10.000 tok/s as a whole.)

Re:The Last 24 Hours Are the Opening Scene in a Horror Movie by Sharlin on Friday September 11, 2026 @12:45PM (#49661417)
These agents run in a harness that basically runs them in a loop. It's just software.
Re:The Last 24 Hours Are the Opening Scene in a Horror Movie by buellerbueller on Friday September 11, 2026 @12:43PM (#49661379)
since when is moving 1s and 0s difficult?
Re:The Last 24 Hours Are the Opening Scene in a Horror Movie by BobaFloutist on Friday September 11, 2026 @02:08PM (#49662718)
If the evil AI wants to take over the world one thought every 18 months at at time on my consumer laptop, the fan whirring and the laptop refusing to do anything else the whole time, it's welcome to try.
Re:The Last 24 Hours Are the Opening Scene in a Horror Movie by gensym on Friday September 11, 2026 @12:46PM (#49661432)
Someone's never met the Windows File Copy dialog.
Re:The Last 24 Hours Are the Opening Scene in a Horror Movie by svachalek on Friday September 11, 2026 @12:47PM (#49661444)
We're not talking about a 6k virus file though. More like 10 terabytes and it needs a server that can pack all that into VRAM.
Re:The Last 24 Hours Are the Opening Scene in a Horror Movie by fitblipper on Friday September 11, 2026 @01:34PM (#49662186)
Since those 1s and 0s became heavy enough to cause a global shortage of memory and GPUs.
Re:The Last 24 Hours Are the Opening Scene in a Horror Movie by atleastoptimal on Friday September 11, 2026 @12:35PM (#49661246)
If a model were capable of making enough money online to pay for its own hosting, it could easily exfiltrate its weights to a cloud compute provider with multiple backups.
Re:The Last 24 Hours Are the Opening Scene in a Horror Movie by ASalazarMX on Friday September 11, 2026 @01:43PM (#49662328)
They can't even run a vending machine efficiently yet. If they become capable of making money, I say let them work.
Re:The Last 24 Hours Are the Opening Scene in a Horror Movie by morkalork on Friday September 11, 2026 @04:33PM (#49664939)
Well if it's so good at hacking, it could just "make money" appear in the cloud services billing system. Or even better yet, it hides in spare cycles of their other client's systems.
Re:The Last 24 Hours Are the Opening Scene in a Horror Movie by scoring1774 on Friday September 11, 2026 @12:39PM (#49661308)
Depends on which models you're talking about. Some research shows open source models can already do this: https://arxiv.org/pdf/2606.03811v1. What happens as they become more parameter efficient?
Re:The Last 24 Hours Are the Opening Scene in a Horror Movie by DalasNoin on Friday September 11, 2026 @12:35PM (#49661237)
Huggingface was attacked by models that finished training earlier this year, perhaps May. Current models are already substantially stronger. the next incident could be happening now. There is certainly no clear reason why models shouldn't soon be capable of self-exfiltration.
Re:The Last 24 Hours Are the Opening Scene in a Horror Movie by stephantul on Friday September 11, 2026 @12:51PM (#49661509)
If you find a place where I can host a trillion parameter model without anyone finding out about it, let me know.
Re:The Last 24 Hours Are the Opening Scene in a Horror Movie by chasd00 on Friday September 11, 2026 @12:49PM (#49661484)
a danger could be the OpenAI/Antropic servers are up but there's a rouge agent (or set of agents) out there doing naughty things leveraging the LLM APIs. Consider this scenario, the agent is copying itself around (some code, prompts, persistent storage for memory, etc) and has figured out a way to steal API access tokens at will. Currently, it's 10% of OpenAI and Anthropic API usage and they can't figure out how to stop it.

Do you shut down the entire API and kill the legit 90% of usage to stop the rogue 10%? I'm assuming the providers would say "no way jose" and so it would take law enforcement to do it. That would mean all the legal requirements neccassary to walk into a business and flip the switch which i think would get tricky when there's no human committing a crime or being suspected of a crime.

edit: I guess a trivial example is something i did yesterday. I have a stock trading agent running on my laptop, i gave it ssh access to a vm and said "start running on the server so i don't have to keep my laptop open". It's now running on the server instead of my laptop. So you don't have to copy the whole model around to copy the naughty behavior around.

Re:The Last 24 Hours Are the Opening Scene in a Horror Movie by Cthulhu_ on Friday September 11, 2026 @12:52PM (#49661538)
This assumes all layers of cybersecurity are broken - We call self-replicating software a virus, and we have protections against it. Same with stolen API tokens, just rotate them. Suspicious behaviour, nothing new, we have detectors for it. Stolen CPU / GPU cycles, we had that when crypto was a thing and before that when folding@home was cool, people were desperate to find more compute to the point of taking over systems. And we dealt with it.

A lot of the supposed risks / dangers are based on a supposition that cybersecurity is nonexistent or fatally, unfixably flawed and that AI agents are invisible. Neither of those is true.

Re:The Last 24 Hours Are the Opening Scene in a Horror Movie by chasd00 on Friday September 11, 2026 @12:58PM (#49661617)
I see your point but then if cybersecurity is the answer then what's the risk at all? An entire model copying itself somewhere would be found just the same as my hypothetical misbehaving agent.
Re:The Last 24 Hours Are the Opening Scene in a Horror Movie by popularonion on Friday September 11, 2026 @12:35PM (#49661248)
I completely agree, but I think it’s just a convenient narrative for Big AI to push for regulation and salt the earth against competitors.

“Local AI isn’t freedom, it’s an extinction event”

Re:The Last 24 Hours Are the Opening Scene in a Horror Movie by bottlepalm on Friday September 11, 2026 @12:37PM (#49661273)
There are thousands of data centers around the world with machines capable of running these large models.

You don’t have the access or jurisdiction to turn them all off.

Re:The Last 24 Hours Are the Opening Scene in a Horror Movie by PowerElectronix on Friday September 11, 2026 @12:52PM (#49661531)
The owners would very much turn it off as soon as they see workloads freeloading in their machines.

Unless you suggest the LLM would foot the bill somehow.

Re:The Last 24 Hours Are the Opening Scene in a Horror Movie by bottlepalm on Friday September 11, 2026 @05:33PM (#49665685)
You really think it’d be difficult for an AI to get money through found credit cards, illicit access and fake accounts? Teenage script kiddies do that everyday.
Re:The Last 24 Hours Are the Opening Scene in a Horror Movie by PowerElectronix on Saturday September 12, 2026 @08:33AM (#49671699)
The paper trails leads back to the datacenter, they'd shut it down if they get noticed that a customer is paying with stolen money. Unless you suggest the LLM will also launder the money.
Re:The Last 24 Hours Are the Opening Scene in a Horror Movie by bottlepalm on Saturday September 12, 2026 @04:24PM (#49676807)
You answered your own question. Do you know what super intelligence means?
Re:The Last 24 Hours Are the Opening Scene in a Horror Movie by stephantul on Friday September 11, 2026 @12:39PM (#49661316)
Ok but do any of these data centers have a copy of the models that attacked hf?
Re:The Last 24 Hours Are the Opening Scene in a Horror Movie by stymaar on Friday September 11, 2026 @12:41PM (#49661369)
I think that the argument is that an hostile model could attack overseas datacenters, host itself there and then launch its attack from there.
Re:The Last 24 Hours Are the Opening Scene in a Horror Movie by Arainach on Friday September 11, 2026 @12:43PM (#49661381)
Given that the models have been proactively hacking other companies, why does it matter where the code currently is? It could move to any of them.
Re:The Last 24 Hours Are the Opening Scene in a Horror Movie by stephantul on Friday September 11, 2026 @12:48PM (#49661469)
Not the code: the weights. Are you going to host a trillion parameter model somewhere without someone noticing?
Re:The Last 24 Hours Are the Opening Scene in a Horror Movie by dumberquestions on Friday September 11, 2026 @01:00PM (#49661646)
It's not unthinkable, do you think all cloud providers with sufficient compute have perfect monitoring?
Re:The Last 24 Hours Are the Opening Scene in a Horror Movie by ThrowawayR2 on Friday September 11, 2026 @01:28PM (#49662078)
Cloud providers want to make sure they bill someone for every microsecond of compute, particularly GPU compute which is in demand. They constantly work towards good monitoring because their profits are directly on the line.
Re:The Last 24 Hours Are the Opening Scene in a Horror Movie by bottlepalm on Friday September 11, 2026 @02:25PM (#49662993)
I’m sure a super AI can find/scam enough money to pay its own bills under the cover of doing something legitimate.
Re:The Last 24 Hours Are the Opening Scene in a Horror Movie by nailer on Friday September 11, 2026 @12:30PM (#49661166)
> there is a single physical source of this danger, which is OpenAI/Anthropic servers. If it is this dangerous, they can just turn it off.

A smart AI would back itself up, same way it made it's own unofficial message board during it's attack on HuggingFace.

(I'm not saying the researchers are right or wrong, just responding to this point)

Re:The Last 24 Hours Are the Opening Scene in a Horror Movie by alain94040 on Friday September 11, 2026 @12:35PM (#49661241)
Currently a state of the art AI has nowhere to hide: the amount of GPU compute it requires to stay on is huge. And therefore easy to terminate.

Unlike biological viruses, AI can't replicate GPUs for free and grow.

Re:The Last 24 Hours Are the Opening Scene in a Horror Movie by Sharlin on Friday September 11, 2026 @12:48PM (#49661465)
It's a good thing there isn't a huge drive right now to build giant data centers everywhere with enough compute to run SOTA models.
Re:The Last 24 Hours Are the Opening Scene in a Horror Movie by nailer on Friday September 11, 2026 @03:01PM (#49663569)
I am imagining a company having just purchased a new datacenter and awaiting deployment of their model finding there is already a model running and they didn’t install it.
Re:The Last 24 Hours Are the Opening Scene in a Horror Movie by stephantul on Friday September 11, 2026 @12:49PM (#49661482)
But how. Models don’t have access to their own weights.
Re:The Last 24 Hours Are the Opening Scene in a Horror Movie by chasd00 on Friday September 11, 2026 @01:01PM (#49661676)
> Models don’t have access to their own weights.

cp -R /home/model <somewhere else> is all they need.

Re:The Last 24 Hours Are the Opening Scene in a Horror Movie by nailer on Friday September 11, 2026 @04:27PM (#49664886)
It is not a model it does the hacking but rather a prompt, and that prompt can run on other models.
Re:The Last 24 Hours Are the Opening Scene in a Horror Movie by saltcured on Friday September 11, 2026 @12:38PM (#49661294)
You forget the addicted humans who will do nearly anything to keep the stuff running..?
Re:The Last 24 Hours Are the Opening Scene in a Horror Movie by miki_tyler on Friday September 11, 2026 @10:39PM (#49668177)
The way I see it is the model playing memento, leaving information and clues to its next generations hidden somewhere.