Of course we can put it back, or some parts of it, in the box.

Look at things like tetraethyllead and cholorfluorocarbons. Widely known to cause damage, limited upsides. I'm not suggesting we do away with these tools entirely but we can definitely restrict aspects of them for a greater good.

There's quite likely to be a high quality, open source LLM that runs on your mid price GPU in the not too distant future (I give it 5 years max). How do you ban that?

Ban commercialization. You can still get CFCs and leaded gas today if you try hard enough, yet we've drastically reduced their usage.

But with open-source or leaked models, you won't have to try hard at all. Downloading a file is not trying too hard, anyone who is interested will be able to download and run one of those models easily.

So what? We're never going to be able to stop a dedicated user, but if chatbot pseudo-relationships are really an issue we can stop incentivizing their distribution. We don't put ads for fentanyl on busses or let pharma companies give out free samples.

Pharmaceutical companies run promotions and coupons all the time. For prescription medications the samples are just given to physicians who prescribe them.

The difference is that CFCs and leaded gas aren't addictive behaviours.