Rapid evolution is well underway. Lone individuals are able to push the envelope of what's possible even with just a new basic interop, maybe in an afternoon. It's much too late to be discussing things like moratoriums.
Maybe such things could prevent emergence when the basics don't exist yet, but not when we're all already walking around holding a capable factory in our hands and can create a new product line in a few lines of Python.
Yes, plenty of low hanging fruit around; Heck, I can probably literally ask chatGPT to implement for me a few ideas I've got.
OTOH, I've known since secondary school of two distinct ways to make a chemical weapon using only things commonly found in normal kitchens, and absolutely none of the post 9/11 aftershock attacks that got in the news over the next decade did anything remotely so simple, so that example makes me confident that even bad rules passed in haste — as many of them were and remain — can actually help.
(And that's despite my GCSE Chemistry being only grade B).
Clearly, it's a very hard problem with massive uncertainties. But we can take actions that will significantly decrease the risk of utter catastrophe.
I don't even think world-ending catastrophe is that likely. But it seems a real enough possibility that we should take it qeriously.
That's simply not true. Nobody would have gotten where GPT is today without transformers. That's not a trivial bit of insight anybody could have had. Stopping research funding and publications will prevent rapid evolution.