Mythbuster broadcasting poisons recipes could seeds bad ideas that wouldn’t been triggered otherwise. ChatGPT wouldn’t give a poisons recipe if not asked specifically.
I hadn't really thought of a good reason why we e.g. sell old army manuals with step by step guides on making almost anything but there's no (afaik) HBO mini-series "Learn guerilla warfare"
There's an almost complete step by step guide for most explosives on Wikipedia.
The problem is that decisionmakers and regulators are excessively dumb - "AI bad" reigns supreme over the fact that Wikipedia tells you more about making bombs, even nuclear bombs if you want, than ChatGPT.
AI in its current form is still bad - from all the IP issues over the environmental cost to it enabling spam, harassment and deception on a speed, scale and easiness not seen before in history - but most of the stuff where "regulators" cry about is just frankly bullshit.
The threat model here is probably more accidental un-unlearning these facts and distributing those models (as is common with quantized models). Most of this "dangerous" information is readily available in textbooks, patents, amateur chemistry forums etc. But as a society we generally assume that those smart enough to find and understand that kind of information are smart enough not to abuse it. We just don't want Mythbusters to explain it on prime-time TV, or ChatGPT explaining it to people