News for hackers. Stuff that matters.
Re:Pause Giant AI Experiments: An Open Letter by noodlesUK on Wednesday March 29, 2023 @05:35PM (#35364076)
I dunno, this sort of scenario really doesn’t worry me too much. There are thousands (maybe tens of thousands) of subject matter experts who could probably develop dangerous weapons like you describe, but none of them seem to just wake up in the morning and decide “today’s the day I’m going to bring the apocalypse”.

I don’t think that this really changes that.

Re:Pause Giant AI Experiments: An Open Letter by olejorgenb on Wednesday March 29, 2023 @05:47PM (#35364226)
I guess the argument would be that the AI machinery will lower the bar, increasing the number of lunatics with the ability to wipe out humanity.
Re:Pause Giant AI Experiments: An Open Letter by root_axis on Wednesday March 29, 2023 @06:12PM (#35364563)
Will it though? Assuming it's even possible for a LLM to e.g. design a novel virus, actually synthesizing the virus still requires expertise that could be weaponized even without AI.
Re:Pause Giant AI Experiments: An Open Letter by tremon on Wednesday March 29, 2023 @06:43PM (#35364929)
Hey GPT-6, construct a floorplan and building instructions for constructing a bioprocess production facility. The building should look like a regular meat packing plant on the outside, but have multiple levels of access control and biohazard management systems.
Re:Pause Giant AI Experiments: An Open Letter by root_axis on Wednesday March 29, 2023 @08:06PM (#35365804)
Let me guess, AI drones to harvest and process the raw materials, construction bots to build the facility, which is of course a fully autonomous bio lab.
Re:Pause Giant AI Experiments: An Open Letter by staunton on Thursday March 30, 2023 @05:25AM (#35370088)
More like Aum Shinrikyo but with an AI as evil mastermind, with brainwashed humans doing its bidding
Re:Pause Giant AI Experiments: An Open Letter by tr_user on Wednesday March 29, 2023 @06:43PM (#35364915)
You're saying they can't make one now. The question is what are we doing before that happens because if you're only thinking about acting when it's viable we're all probably already dead.
Re:Pause Giant AI Experiments: An Open Letter by Teever on Wednesday March 29, 2023 @06:41PM (#35364902)
What if you ask the LLM to design a simplified manufacturing process that could be assembled by a simple person?

What if you ask the LLM to design a humanoid robot that assemble complex things, but could be assembled by a simple person?

Re:Pause Giant AI Experiments: An Open Letter by root_axis on Wednesday March 29, 2023 @08:12PM (#35365859)
LLMs aren't magic, the knowledge of how to design a humanoid robot that can assemble complex things isn't embodied in the dataset it was trained on, it cannot probe the rules of reality, it can't do research or engineering, this knowledge can't just spontaneously emerge by increasing the parameter size.
Re:Pause Giant AI Experiments: An Open Letter by wizzwizz4 on Wednesday March 29, 2023 @06:45PM (#35364947)
I could synthesise this theoretical virus the computer spat out, that may or may not be deadly (or even viable). Or I could download the HIV genome from the arXiv, and synthesise that instead.

(Note: as far as I can tell, nobody's actually posted HIV to the arXiv. Small mercies.)

Re:Pause Giant AI Experiments: An Open Letter by dekhn on Wednesday March 29, 2023 @07:08PM (#35365196)
The sequence of HIV is published and has been for a very long time. In fact there's a wide range of HIV sequences: https://www.ncbi.nlm.nih.gov/Taxonomy/Browser/wwwtax.cgi?id=...

You could synthesize that genome but it wouldn't be effective without the viral coat and protein package (unlike a viroid, which needs no coating, just the sequence!).

I should point out that in gene therapy we use HIV-1 derived sequences as transformation vectors, because they are so incredibly good at integrating with the genome. To be honest I expected work in this area would spontaneously and accidentally (or even intentionally) cause problems on the scope of COVID but (very fortunately) it never did.

One would like to be able to conclude that some virus work is inherently more safe than other virus work, but I think the data is far to ambiguous to make such a serious determination of risk.

Re:Pause Giant AI Experiments: An Open Letter by KidComputer on Wednesday March 29, 2023 @05:49PM (#35364251)
I'm willing to wager there are zero subject matter experts today who could do such a thing. The biggest reason is that the computational methods that would let you design such a thing in-silico are not there yet. In the last year or two they have improved beyond what most people believed was possible but still they need further improvement.
Re:Pause Giant AI Experiments: An Open Letter by csnover on Wednesday March 29, 2023 @06:01PM (#35364420)
I am not a subject expert here at all so I don’t know if I understand exactly what you mean by “methods that would let you design such a thing in-silico”, but there was a paper[0] and interview with its authors[1] published a year ago about a drug-development AI being used to design chemical weapons.

[0] https://www.nature.com/articles/s42256-022-00465-9

[1] https://www.theverge.com/2022/3/17/22983197/ai-new-possible-...

Re:Pause Giant AI Experiments: An Open Letter by digdugdirk on Wednesday March 29, 2023 @06:07PM (#35364503)
I see the major issue with AI as one of "lowering the bar".

For example - I'm a mechanical engineer. I took a programming class way back in university, but I honestly couldn't tell you what language was used in the class. I've gotten up to a "could hack a script together in python if need be" level in the meantime, but it comes in fits and spurts, and I guarantee that anyone who looked at my code would recoil in horror.

But with chatGPT/copilot covering up my deficiencies, my feedback loop has been drastically shortened, to the point where I now reach for a python script where I'd typically start abusing Excel to get something done.

Once you start extending that to specific domains? That's when things start getting real interesting, real quick.

Re:Pause Giant AI Experiments: An Open Letter by jupp0r on Wednesday March 29, 2023 @06:19PM (#35364642)
You confuse syntax with semantics. Being able to write produce good quality small snippets of python will not enable you to produce a successful piece of Software. It's just an entirely different problem. You have to unterstand the problem, the environment in which it exists to create a good solution. ChatGPT doesn't (as of now).
Re:Pause Giant AI Experiments: An Open Letter by lanstin on Wednesday March 29, 2023 @07:29PM (#35365422)
I had hoped to have ChatGPT do my work today, but even after a number of iterations it was having compiler errors and referring to APIs not in the versions it was having me install.

A bit different from stack overflow, but not 10x. It was flawless when I asked it for syntax, e.g. a map literal initializer in Go.

On the other hand, I asked it to write a design for the server, and it was quite good, writing more quantity with and more clarity than I had written during my campaign to get the server approved. It even suggested a tweak I had not thought of, although that tweak turned out to be wrong it was worth checking out.

So maybe heads down coding of complex stuff will be ok but architects, who have indeed provided an impressive body of training data, will be replaced. :)

Re:Pause Giant AI Experiments: An Open Letter by digdugdirk on Wednesday March 29, 2023 @09:24PM (#35366500)
That's the thing though, it is successful. To my exact needs at the moment. It's not necessarily reliable, or adaptable, or useful to a layperson, but it works.

Getting from "can't create something" to "having something functional and valuable" is a huge gap to leap over, and as AI is able to make those gaps smaller and smaller, things are going to get interesting.

Re:Pause Giant AI Experiments: An Open Letter by tr_user on Wednesday March 29, 2023 @06:41PM (#35364895)
I think you're very wrong about this. I think this is similar to gun control laws. A lot of people may have murderous rage but maybe the extent of it is they get into a fist fight or at most clumsily swing a knife. Imagine how safe you'd feel if everyone in the world was given access to a nuke.
Re:Pause Giant AI Experiments: An Open Letter by haxiomic on Wednesday March 29, 2023 @07:40PM (#35365551)
If everyone had an app on their phone with a button to destroy the world the remaining lifetime of the human race would be measured in milliseconds

Now if this button was something you had to order from Amazon I think we’ve got a few days

There’s a scenario where people with the intent will have the capability in the foreseeable future

Re:Pause Giant AI Experiments: An Open Letter by fullstackchris on Wednesday March 29, 2023 @07:55PM (#35365717)
like what? would you rather have a gpt5 or a nuke? pure fearmongering. what am i gonna do, text to speech them to death? give me a break
Re:Pause Giant AI Experiments: An Open Letter by haxiomic on Wednesday March 29, 2023 @08:06PM (#35365810)
Here’s someone who orders parts from the internet to design a custom virus that genetically modifies his own cells to cure his lactose intolerance https://youtu.be/aoczYXJeMY4

Pretty cool for sure and a great use of the technology. The reason more of us don’t do this is because we lack the knowledge of biology to understand what we’re doing

That will soon change.