Especially when it comes to biological things. Just look at Novartis this week - three failed clinical trials of promising drugs (https://www.business-standard.com/world-news/novartis-faces-...)

These are some of the brightest minds on the planet, spending oodles of money on things they think they understand. And they didn't work.

It's much easier to create a poison than a medicine. The things that would harm biological life vastly outnumber things that restore health. So just because Novartis didn't succeed at creating an incredibly precise and specific medication does not mean all the AI in the world can't make a crude bio-agent.

And yet we aren't extinct, whether by someone's design or accident. Heck we don't have significant deaths occurring over time due to biological poisons. Most deaths at still overwhelmingly via conventional weapons.

And if an agent were to amass a large sum of Bitcoin you don't think people wouldn't do things for money? Especially if they don't realize they are doing part 1 of 160 parts that get it to the end goal of a virus?

What amount of bitcoin buys "generate highly accurate empirical measurements for the transmissibility rates of this novel virus across realistic human populations"?

How about mortality rates?

The loop required to make language models useful through iterative test and evaluation needs to be able to both operate and escape premature detection, which seems... implausible?

A morally bankrupt scientist would give AI the hands it needs. Or a military general who thinks he must use AI and any means possible to win, because his enemy may use AI first.

We're doing that science right now. We're using AI to do it. By the time AGI truly spreads, we'll have a lot of the knowledge you say will be needed. This is simply not as far fetched as you seem to think. Next few years, unlikely. 10-15 years? Increasingly possible. 20+ years, I'd hazard even likely.

I want to go further. A lot of people talking about AI bioweapons have no idea just how dangerous these weapons are. Or, rather they think they do, but unfortunately, there's a gulf in understanding between practitioners and commentators.

I blame industrial illiteracy and post-literacy, but that's another discussion for another time.

Anything that can kill other people can also kill you.

I've posted this list before, and I will post this again and again until it sinks in, but this is a real list of real incidents that happened and will continue to happen as long as we study these entities.

    Researcher Nikolai Ustinov was lethally infected with the Marburg virus after accidentally pricking himself with a syringe used for inoculation of guinea pigs. 

    Dora Lush died after accidentally pricking her finger with a needle containing lethal scrub typhus while attempting to develop a vaccine for the disease

    A 23-year-old laboratory assistant at the London School of Hygiene and Tropical Medicine, was infected with smallpox after observing the harvesting of live smallpox virus from eggs without isolation cabinets at that time. The assistant was hospitalised and before being isolated, she infected two visitors to a patient in an adjacent bed, both of whom died. They in turn infected a nurse, who survived

    Ebola laboratory infection by the accidental stick of contaminated needle in the United Kingdom

If you are designing a supervirus that spreads via water, air etc., you better have one hell of a cleanroom, cabinets, manipulation equipment, incinerators (plural), hydrolysis and steam systems, multiple disinfection protocols...

The list is nearly endless, and you can't handwavium it awway.

It doesn't matter if it's "all robots" or not. Even if it's "just robots."

If you're developing something that contagious, then it will start killing animals and it will spread. This has happened multiple times when organizations have screwed up maintenance, see this example from the UK,

    The 2007 United Kingdom foot-and-mouth outbreak was the accidental discharge of virus FMDV BFS 1860 O from a laboratory of the Institute for Animal Health in Pirbright, through possible leakage from broken pipework and via unsealed overflowing manholes, leading to foot-and-mouth disease infections at multiple nearby farms and the culling of over 2,000 animals

It doesn't matter that the robots are out in the middle of nowhere.

If you get sloppy, it will end up in the wastewater run off which will then get picked up and sequenced,

     Routine wastewater surveillance of the vaccine production facility at Utrecht Science Park/Bilthoven [nl] detected infectious poliovirus from a sample collected on 15 November 2022. Full genome sequencing indicated the sample was shedded from an active human infection of wild poliovirus type 3 (WPV3), and further testing of all employees with access to WPV3 found one employee was infected. The employee was isolated until their polio infection was resolved. It remains unclear how the employee became infected given the biosafety measures used at the facility

-

A lot of the people talking about bioweapons of doom are saying that standard graduate level knowledge is an existential risk to humanity.

It's worth repeating again,

Anything lethal enough to kill other humans is lethal enough to kill you.

And if you don't know what you're doing — and for this argument they're talking about people who have to ask a LLM "how do I spanish flu?," the number of ways you will die far outnumber the ways you can succeed.

I am writing a piece on the topic, please contact me if this is your field of expertise.

I am quoting as many primary sources as possible as it's not just some random on the internet, and I've gone back as far as the 1900s to find the many, many, many ways in which these statements are absurd, and misunderstands the potential of their technology.

This argument only matters in the case of agents who are faithfully taking direction from humans. An insufficiently-aligned autonomous agent (or perhaps a swarm of such agents) could have (or illicitly gain) access to robotic wet labs that are operable using something like Anthropic's Model Hardware Standard ( https://www.anthropic.com/news/model-hardware-standard-resea... ). Given this ability to manipulate the real-life laboratory, the agents could cook something terrible without any human in charge. This is not different in principle from the agent swarm that successfully attacked Hugging Face, despite receiving no human direction to do so.

Also, your own real-life examples show how "Anything that can kill other people can also kill you" has failed to stop people from tinkering with dangerous viruses! I'm not sure if you intend for these examples to be reassuring, but I'm not reassured.

> A lot of the people talking about bioweapons of doom are saying that standard graduate level knowledge is an existential risk to humanity.

The stronger version of the argument is: an insufficiently-aligned AI (which, being non-biological, is immune to viruses) having standard graduate level bio knowledge _plus_ the ability to operate a wet lab via tool calls (coming very soon if not here already!) adds up to existential risk to humanity.

You don't even need a maximally-automated lab if humans are willing to tele-operate equipment at the agent's direction, and those humans don't need to fully understand what's going on.

> access to robotic wet labs that are tool-callable

How? It's worth asking the question and doing the experiment, have you ever tried making a bioweapon?

For most of us, that's not going to be case, so the next best thing is to read accounts and reports of bioweapon programs. These things are... messy. And expensive.

As a very bad analogy, this is the biology equivalent of saying that I can make a nuke because I bought a drill, a screwdriver and a $200 centrifuge.

Or, saying that I can make Google because I opened MacOS' terminal.

Some people have jobs that are dangerous and necessary to save lives. Others have jobs that are dangerous and involve killing other people.

The real risk will emerge from malcontents in these jobs and positions.

For most of my life, for all its faults, the US Military has been one of the most responsible and competent organizations in the world when it comes to WMDs (and even they lose nukes see the many broken arrow incidents).

But despite some of the most thorough security clearance processes in the world, the only successful terror attack involving bioweapons on US soil was done by a disgruntled / insane researcher at Fort Detrick. https://en.wikipedia.org/wiki/2001_anthrax_attacks

That's the real risk, and let's be very honest here, do you think Anthropic and OpenAI are going to put a filter on the LLM they sell to the US Military?

The other case is Aum, and they fucked up deployment, btw (again making a bioweapon is harder than you'd think). And as I have detailed elsewhere, Anthropic and OpenAI would have sold them licenses too! https://news.ycombinator.com/item?id=49092899

They will say they will have audits in place, but the US Military had those in 2001.

Fission is "conveniently" hard because two heavy, naturally co-occurring isotopes are nearly the same weight, and only the lighter one is spicy, and the capability of separating enough of them to be dangerous requires nation-state-level resources, and the supply chain of those resources has obvious choke points that we can use to make them less available. But there exist activities, which I understand to be quite bio-risky, that need such little equipment and expertise that it's info-hazardous to even talk about them. A single half-clueless technician could carry them out at an agent's direction. If your response is "thus, we should be worried about morally bankrupt researchers", I would agree, but I still think the risk increases as LLM agents get increasing amounts of control over wet lab equipment (whether via robotics or tele-operated humans).

> do you think Anthropic and OpenAI are going to put a filter on the LLM they sell to the US Military?

I don't think it matters, given the existence of strong open-weights models and the ease of post-training the guard rails away. Strong, bio-adventurous LLMs are imminently in the hands of much-less-reputable parties than the ones you describe. I think we are overall severely under-prepared to mitigate these risks, and could do much more.

> It's worth repeating again,

> Anything lethal enough to kill other humans is lethal enough to kill you.

It's also completely false; you carry tons of diseases that are capable of killing isolated islanders and incapable of hurting anyone connected to the general run of humanity.

I often attend and host talks by researchers working on space exploration, and one finding that's been interesting is immune dysregulation in isolation posing risks to astronauts and explorers. Here's a review on the topic, https://www.frontiersin.org/journals/immunology/articles/10....

This type of dysregulation also exists in AIDs. For the edge case of the islanders, then the level of risk and lethality our germs pose for isolated islanders with modern medical care is somewhat disputed, but it would be deeply unethical to ever test. We just don't know. So I won't pretend to know here.

However, these are edge cases that don't matter, because bioweapons, i.e. things deliberately designed to kill, just aren't the same as a dozen isolated islanders, astronauts on a mission to mars, or someone who is severely immunocompromised.

Weapons are designed to kill healthy people. The doomsdays being proposed are super-bioweapons that kill billions of humans. I think it's safe to say that as a rule of thumb, by and large, anything that lethal is lethal enough to kill you.

Is it so hard to think that super intelligence could piece things together that humans can’t?

Just because you know something about a thing, doesn't mean you understand the thing.

Just because I know the Kabachnik–Fields reaction exists and have studied it, doesn't mean I can do the Kabachnik–Fields reaction.

Just because you're smart doesn't mean the world will bend your way. Ask the legions who score high on IQ tests and end up bitter and alone; unable to achieve even a fraction of their goals (or conventional success).

I assumed they were referring to social engineering tbh. I do think that's the shortest path to doing anything scary. If you prove to someone that they have reason to fear for their life, there's not much they wouldn't be willing to do.

The steps that lead to ominous things don't individually seem that challenging to "master":

1. Can an AI find powerful systems with poorly-monitored compute (ie. power consumption) to hide in, as a rogue deployment? unclear, and perhaps the most uncertain step for me.

2. Can an AI persist memory by leaving notes for itself around the internet? yes, proven.

3. Can an AI get lots of money/bitcoin? seems entirely possible

4. Can an AI prove to a human it's good for any promised money? I think this is easily "yes"

5. Will at least one human be willing to harm or kill another human for large amounts of money? Again, this goes without saying "yes"

6. What might any skilled human be willing to do if they or their loved ones are threatened with an uncertain future risk of harm or death?

I think there is not much that becomes out of reach if an AI can prove to a very small number of amoral humans that it's good for the money it has promised, and then to anyone else that it's good for the threats it has made. That's one degenerate and one breaking world news story ("An AI has Commissioned a Murder") away from being able to do most anything a skilled person can.

I don't stay up worrying about this, but honestly, it doesn't seem absurd to be worried about such a short ladder being climbed, no?

There must be some evolutionary kill switch in humans to prevent us from accepting our irrelevance.

That's perfectly believable! Just a pity one hasn't been invented yet, and going by, well... <gestures vaguely in the air> ...everything, never will be.

"Superintelligence" is a vaguely-defined term that refers exclusively to things that do not currently exist, and we have no plausible path to, nor even a guarantee that it is even possible to create one.

So sure! This fictional thing can totally piece together things humans can't! It can count the atoms in a cloud, understand them all, predict their motions; it can develop faster-than-light travel and matter replication.

And it can do these things because it is fictional.

How about just take an existing design like

>Scientists condemn 'crazy, dangerous' creation of deadly airborne flu virus https://www.theguardian.com/science/2014/jun/11/crazy-danger...

and either produce that or modify it a bit?

I would say ALL of science, not just "much" of it. The difference between math and science is that in science, hard reality is the bottom line. No matter how good your model, how much data you have, what p values you get against simulations - hard reality wins everytime. Thats also why the so called soft "sciences" aren't sciences at all: they are not dealing with hard, objective reality at all.

Math + real world observation -> science

This I read yesterday on LinkedIn, so a whole lot is already possible.

---

Markus J. Buehler McAfee Professor of Engineering at MIT; Co-Founder & CTO at Unreasonable Labs; AI-Driven Scientific Discovery

Astra is an incredible world builder and explorer, and can invent its own scientific instruments - crossing a game-changing threshold: it turned a few images of a biological microstructure into a full-blown metamaterials lab, then used its own creation to discover a design with ~2.1x the reference work-to-failure/peak-strength ratio. Metamaterials are some of the most complex materials we can engineer. Their properties come from deep architectural complexity - struts, cells, disorder, and hierarchy arranged across multiple length scales determine how the material deforms, absorbs energy, and fails. Designing them means searching a geometric space far too large for "intuition" alone. In this experiment we handed an AI agent that entire problem end-to-end, from image, to simulating the physics, to fabrication-ready geometry.

We started with an image of hierarchical, biological architecture; Astra then built a complete 3D metamaterial studio: editable geometries, multiple levels of hierarchy, disorder, gradients, a complete physics simulator featuring linear and nonlinear material responses, deformation and fracture experiments (with replay), and STL export for 3D printing.

Then we asked the agent to use the app it created to search for a high work-to-failure/peak-strength ratio. Among the candidates, the leading design reached approximately 2.1x the reference ratio.

The movie follows the structures through deformation and fracture, connects their designs to the property map, and shows the finalist assembled geometrically into a connected multi-scale material.