We should stop with this question. It is never in good faith and I always see it presented in very weird ways. As a matter of fact, we have an answer to this already, but maybe it hasn't explicitly been said (enough). We've decided that the answer is "yes." In fact, I can't think of a way that this can't be yes because otherwise how would you make bombs and kill people?

They always have vague notions like can an LLM teach you how to build "a bomb." What type of bomb matters very much. Pipebomb? I don't really care. They sold instructions at Barnes and Nobles for years and The Anarchist Cookbook is still readily available. High explosives? I'd like to see how it can give you instructions where someone that does not already have the knowledge to build such a device could build the device __and__ maintain all their appendages.

A common counter argument to these papers is about how the same information is on the internet. Sure, probably. You can find plans for a thermonuclear weapon. But that doesn't mean much because you still need high technical skill to assemble it. Even more to do so without letting everyone in a 50km radius know. And even harder without appearing on a huge amount of watch lists or setting of our global monitoring systems. They always mask the output of the LLM, but I'd actually like to see it. I'm certain you can find some dangerous weapon where there are readily available instructions online and compare. Asking experts is difficult to evaluate as a reader. Are they answering that yes the instructions are accurate in the context that a skilled person could develop the device? Is that skill level at such a level that you wouldn't need the AI instructions? I really actually care just about the novices, not the experts. The experts can already do such things. But I would honestly be impressed if a LLM could give you detailed instructions on how to build a thermonuclear weapon that was actually sufficient to assemble and not the instructions that any undergraduate physics student could give you. Honestly, I doubt that such instructions would ever be sufficient through text alone and the LLM would have to teach you a lot of things along the way and force you to practice certain things like operating a lathe.

It would also have to teach you how to get all the materials without getting caught which is how we typically handle this issue today: procurement.

I just don't have confidence that this presents a significant danger even if the AI was far more advanced or even were it AGI. An AGI robot operating independently is a whole other story, but I'd still be impressed if it could build a thermonuclear weapon or biological weapon and do so without getting caught. If to teach someone how to build a nuclear weapon you first need to give them a physics and engineering degree, then I'm not worried.

So I do not find these arguments worth spending significant time thinking about. There are much higher priority questions at play and even more dangers from AI than this that are worth spending that time on. Maybe I've characterized incorrectly, but I'm going to need some strong evidence. And if you're going to take up this challenge, you must consider what I've said carefully. About the gap between theory and practice. If you have no experience making things I doubt you will be able to appropriately respond but you're more than welcome to give it a go. If you want a proof by example: take anything you think you know how to do but haven't done before, and then try to go do that. If you fail, we've made my case. If you succeed, consider if you could have done so through instructions alone and what were the underlying skills you needed to bridge the gap.