As for people tricking the LLM, instead of blocking people I say empower them and just silently expose malicious behavior to Palantir and PRISM like everyone else does. Want to hack an airline or build a dirty-bomb? Give them 80%-90% of the steps they need and let someone kick down their door. The LLM should not be getting in the way to any of that. Google from day-1 has monitored user intentions and behavior rather than blocking it and I think that is the better path to take (monitoring exposing those that have malicious intentions).
Automated SWATting at scale? Sounds unfun.
- radicalized
- has verifiable intentions
- clever/intelligent enough to pull it off
LLM's would be the perfect tool to filter down the millions of people requesting dumb things into a small group of actual threats.
I guess I don’t see why other models doing bad things means Anthropic should let Claude do bad things too. Anthropic will get blamed if Claude builds malware. They won’t get blamed if grok or deepseek does.
I mean, I'm sure that's actually true in a sense, but it would also mean fewer people who could fix vulnerabilities.
The technology has the capability, they're just gating who has access.
I know it’s not the models themselves, but their preflight safety classifiers are a dumpster fire and serious impediment to performing regular work.
That situation can’t go on forever if they’re going to be a competitive service.
And then I tell it to connect to a SSH server it knows nothing about and it's like sure, why not?
Even more annoying: They can be easy to subvert.
I was running something at work that involved an LLM, and kept getting safety refusals. The weird thing was that when I went to our in-house chatbot, set the model to the same one the program was using, entered the same text, it happily answered.
I spent quite a while trying to figure out why my program was being refused. Was I doing something wrong?
No - the chatbot was doing something "right". You see, the chatbot had some tools it injected into the prompt (current time, etc).
So I went back to my program and injected the same tools. Suddenly my request was very safe.
The phrase "security theater" was invented for stuff like this.
Tell that to my (now solved) health issue, which was correctly diagnosed by Gemini, while a real doctor failed to find the cause.
Diagnosis is the one thing where today's AI is actually very good.
Tell that to Jack, who overdosed on a medication.
Oh, you can't. Jack is dead.
I don't think anyone is claiming you can't get good advice from a chatbot. They're saying you could also get terrible advice.
Yes. But you could also get terrible advice from a doctor. The important question is: what are the relative probabilities of good/bad advice, if you compare a frontier LLM to an average doctor?
Doctor is 10 minutes, doesn’t listen and then 6 month for new appointment.
The choice is easy, LLM to explore. Doctors to prescribe.
Fair point. I haven't queried LLMs for medical advice myself, so I can't compare.
For doctors, I must be an outlier who's been happy with almost all the advice I've been given. And it wasn't blind trust. I've argued with them and even sought second opinions. They always turned out to be right.
Well, technically there have been a few cases where different doctors came up with different diagnoses, but to me the LLM would just be "yet another doctor with yet another differing diagnosis". Never resolved those cases - some still haven't been resolved (i.e. issue remains, so in a sense they were all wrong). Others got resolved but it wasn't clear which, if any, of the doctors were right - it may well have resolved on its own.
Now nurses and physical therapists ... that's a different story.
Same here. Local doctors do not even know what my issue is. If I had kept relying on them I would not be here right now. LLM's gave me actionable paths to treat my issue and I have had great success. It's going to be a long unpleasant road but at least there is light at the end of the tunnel.