We have medical trials for a reason. It doesn't matter what the drug discovery process is as long as you test them after discovering them.

Testing CRISPR genome edits of pathogens is playing with fire. I hope it takes a small incident for someone to pull the plug on all these moronic biolabs rather than a big one. Or else the $1000 smallpox hell will be a reality quite soon.

> Testing CRISPR genome edits of pathogens is playing with fire

This has nothing to do with the article

It has everything to do with not dying though.

Agree. Personally I'm strong on "AI has real risks", which are mostly inflated beliefs in what it tells us. The protocols for drug testing are so high AI (if it works) is most likely to reduce prescreening delay, and if it had a risk it would be ignoring novel forms which don't conform to training, not high lethality: the drug test regimen deals with that already.

Unless it includes mRNA which tricks your cells into making a highly contagious virus which kills a year after infection... then it just takes a single test. Which probably assumes a perfectly malicious superintelligence and is still well within the realm of science fiction... but even trialling a drug in mice is ultimately "testing in prod."

A whole virus worth of mRNA would be a lot of mRNA (though, a prion wouldn't be) and I think we'd notice it happening.

I'm assuming you mean 'We' as the US. Assuming everyone holds high standards may be a mistake.

Am not aware of any non-US countries having issues with this either.

The US is actually too strict, which is why we don't have any of the good sunscreens.

One could imagine a sufficiently advanced AI coming up with drugs which are designed to appear harmless in drug trials despite still being dangerous, either because it legitimately wants to murder us or because its learning function will be incredibly biased towards passing drug trials. I mean, we already sometimes do that (see the Thalidomide Tradgedy), but presumably AI can do it faster and more efficiently in ways that we would struggle to anticipate.

Open source genomes of engineered super viruses combined with garage-grade nucleotide synthesizers will with more probability than anything else I can think of.

Or, heck, maybe the super virus escapes from a BSL4 lab on accident before the garage phase. :D (There's been precedent, and I'm not alluding to COVID.)

What precedents?

I don't think it was a modern BSL-4 standard lab, but smallpox escaped and killed someone in the UK. Now (at least in theory) only the US and Russia have samples in special government BSL-4 labs. Other countries could make it from sequence though.

Edit: I can't reply, so I'll say that an extinct horsepox virus was recreated from sequence, the same procedure in theory should work on smallpox

Smallpox is 186 kilobases in size. It's pretty hard to synthesize from scratch.

Not impossible, but hard.

Then you'll still need to assemble the viable viral particles. This will probably require the creation of artificial chromosomes needed for the viral replication, and then innoculating human cell culture with them, alongside with the synthetic viral DNA.

This is a level that requires years of work from a major biolab.

I dunno, these guys just ordered fragments from GeneArt: https://journals.plos.org/plosone/article/file?id=10.1371/jo...

The did have a BSL2 lab, and they did quite a bit of work. But yeah, the state of the art has advanced since I worked in molecular biology.

Always harder than you think of course - but the thing about molecular biology is unlike nuclear physics, most of the equipment you need is relatively cheap and readily available.

So the tools are there.

However on the other hand - making something the right mix of lethal, but still able to spread I suspect is incredibly hard - if it was easy we'd all be dead already ( from viruses etc naturally evolving ).

Russia definitely doesn't have BSL-4 labs (because BSL-4 refers to an US-specific system).

Sorry for being pedantic.

>I don't think it was a modern BSL-4 standard lab, but smallpox escaped and killed someone in the UK.

That was in 1978! I believe we didn't have biosafety levels back then in the UK.

Edit: at least the BSL levels came 6 years later, but the UK has different names for those

>Over the next two decades, growing CDC, NIH, and OSHA participation in ABSA annual meetings further solidified biosafety guidelines, culminating in the 1984 publication of the first edition of the text, Biosafety in Microbiological and Biomedical Laboratories (BMBL). The BMBL guidelines laid out four levels of increasingly intensive safety practices, equipment, facilities, and engineering controls to be employed in the safe handling of microbial agents: Biosafety Levels 1, 2, 3, and 4 (BSL-1, -2, -3, and -4),

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7099915/

You can acquire viable smallpox samples from graves in permafrost regions. It would not surprise me if the UK incident was an archeologist.

This is a good list on BSL incidents:

https://en.wikipedia.org/wiki/List_of_laboratory_biosecurity...

Lots. I think one of the only arguments that needs to be made against things like gain-of-function research is this. [1] That's an incomprehensive list various announced biosecurity incidents.

Just since 2000 we've had lab incidents, including leaks, with: anthrax, west nile virus, SARS, COVID (not implying China - there was a confirmed COVID leak in Taiwan), ebola, tuberculosis, dengue, smallpox, zika, polio, and more. And they're happening all throughout the world. That includes the US, China, Russia, Japan, Germany, Australia, UK, South Korea, Hungary, France, Taiwan, Netherlands, and more. Incidents specified as coming from BSL-4 labs include ebola and SARS, though the BSL level is not specified at all for most incidents.

And I would take that is an extremely incomprehensive list given that many incidents are likely going to be tucked away or classified. There's playing with fire, and then there's this... which increasingly more feels like standing around a fire and seeing what happens if you start dumping kerosene into it, all in the name of firefighting - of course.

[1] - https://en.wikipedia.org/wiki/List_of_laboratory_biosecurity...

I love how the Longtermism people won't shut up about this without having one single clue what they're actually talking about.

As though virus particles spontaneously pop into existence when you mix the right DNA strands together (a process that gets wildly expensive beyond a couple hundred basepairs and is destroyed if the proteins on your fingers get anywhere near it).

That's notwithstanding the fact that you can't just turn a dial labelled "lethality" to produce a gene sequence.

The expense seems to have gone down enough in recent years that there are too many sketchy biolabs mixing a lot of base pairs together. Or doing other kinds of things to cross breed these bugs.

Yes, if AI can do this, expect AI to also find diseases that are perfect bioweapons...such as a long-lasting common cold that merely weakens you for months, and is extremely contagious. Perfect for weakening an entire country before invasion, without even a hint that it actually is a bioweapon until it's too late.

Weapons that attack the entire planet are not perfect.

They will be when our robot overlords want to get rid of us. :-)

This assumes robots would want to be friend with each other, because robots. They'd probably wipe each other out very efficiently too and they would probably be able to do it without the use of weapons, it would just be cyber warfare.

No weapon in the history of mankind has ever been perfect.

Loads of things can be used as inefficient weapons, for example we could be worried about rouge states developing methods of dropping dead cows from zeppelins.

You jest, but launching dead cows from catapults was indeed a medieval siege tactic.

How? Not snark, genuine question. What do you see about this form of active molecule search that makes it uniquely dangerous?

It seems very close to resurrecting ancient diseases and protein strands 'just because we can'.