This argument only matters in the case of agents who are faithfully taking direction from humans. An insufficiently-aligned autonomous agent (or perhaps a swarm of such agents) could have (or illicitly gain) access to robotic wet labs that are operable using something like Anthropic's Model Hardware Standard ( https://www.anthropic.com/news/model-hardware-standard-resea... ). Given this ability to manipulate the real-life laboratory, the agents could cook something terrible without any human in charge. This is not different in principle from the agent swarm that successfully attacked Hugging Face, despite receiving no human direction to do so.

Also, your own real-life examples show how "Anything that can kill other people can also kill you" has failed to stop people from tinkering with dangerous viruses! I'm not sure if you intend for these examples to be reassuring, but I'm not reassured.

> A lot of the people talking about bioweapons of doom are saying that standard graduate level knowledge is an existential risk to humanity.

The stronger version of the argument is: an insufficiently-aligned AI (which, being non-biological, is immune to viruses) having standard graduate level bio knowledge _plus_ the ability to operate a wet lab via tool calls (coming very soon if not here already!) adds up to existential risk to humanity.

You don't even need a maximally-automated lab if humans are willing to tele-operate equipment at the agent's direction, and those humans don't need to fully understand what's going on.

> access to robotic wet labs that are tool-callable

How? It's worth asking the question and doing the experiment, have you ever tried making a bioweapon?

For most of us, that's not going to be case, so the next best thing is to read accounts and reports of bioweapon programs. These things are... messy. And expensive.

As a very bad analogy, this is the biology equivalent of saying that I can make a nuke because I bought a drill, a screwdriver and a $200 centrifuge.

Or, saying that I can make Google because I opened MacOS' terminal.

Some people have jobs that are dangerous and necessary to save lives. Others have jobs that are dangerous and involve killing other people.

The real risk will emerge from malcontents in these jobs and positions.

For most of my life, for all its faults, the US Military has been one of the most responsible and competent organizations in the world when it comes to WMDs (and even they lose nukes see the many broken arrow incidents).

But despite some of the most thorough security clearance processes in the world, the only successful terror attack involving bioweapons on US soil was done by a disgruntled / insane researcher at Fort Detrick. https://en.wikipedia.org/wiki/2001_anthrax_attacks

That's the real risk, and let's be very honest here, do you think Anthropic and OpenAI are going to put a filter on the LLM they sell to the US Military?

The other case is Aum, and they fucked up deployment, btw (again making a bioweapon is harder than you'd think). And as I have detailed elsewhere, Anthropic and OpenAI would have sold them licenses too! https://news.ycombinator.com/item?id=49092899

They will say they will have audits in place, but the US Military had those in 2001.

Fission is "conveniently" hard because two heavy, naturally co-occurring isotopes are nearly the same weight, and only the lighter one is spicy, and the capability of separating enough of them to be dangerous requires nation-state-level resources, and the supply chain of those resources has obvious choke points that we can use to make them less available. But there exist activities, which I understand to be quite bio-risky, that need such little equipment and expertise that it's info-hazardous to even talk about them. A single half-clueless technician could carry them out at an agent's direction. If your response is "thus, we should be worried about morally bankrupt researchers", I would agree, but I still think the risk increases as LLM agents get increasing amounts of control over wet lab equipment (whether via robotics or tele-operated humans).

> do you think Anthropic and OpenAI are going to put a filter on the LLM they sell to the US Military?

I don't think it matters, given the existence of strong open-weights models and the ease of post-training the guard rails away. Strong, bio-adventurous LLMs are imminently in the hands of much-less-reputable parties than the ones you describe. I think we are overall severely under-prepared to mitigate these risks, and could do much more.

> It's worth repeating again,

> Anything lethal enough to kill other humans is lethal enough to kill you.

It's also completely false; you carry tons of diseases that are capable of killing isolated islanders and incapable of hurting anyone connected to the general run of humanity.

I often attend and host talks by researchers working on space exploration, and one finding that's been interesting is immune dysregulation in isolation posing risks to astronauts and explorers. Here's a review on the topic, https://www.frontiersin.org/journals/immunology/articles/10....

This type of dysregulation also exists in AIDs. For the edge case of the islanders, then the level of risk and lethality our germs pose for isolated islanders with modern medical care is somewhat disputed, but it would be deeply unethical to ever test. We just don't know. So I won't pretend to know here.

However, these are edge cases that don't matter, because bioweapons, i.e. things deliberately designed to kill, just aren't the same as a dozen isolated islanders, astronauts on a mission to mars, or someone who is severely immunocompromised.

Weapons are designed to kill healthy people. The doomsdays being proposed are super-bioweapons that kill billions of humans. I think it's safe to say that as a rule of thumb, by and large, anything that lethal is lethal enough to kill you.