That’s not entirely true, as it’s currently impossible to actually gauge the severity of what the LLM seemingly enabled you into doing. There’s a difference between “I uncritically accepted everything it told me because it lined up with what I was hoping to hear” and “it subtly nudged me towards a course of action that was going to be obviously unwise after some consideration, but managed to convince me to skip this”; and also between that and “I took a risk, which I knew to be a risk, and which I knew to potentially expect to go bad, and the LLM convinced me to take it where I otherwise wouldn’t have”, and ALSO between that and “I took a risk, which I knew to be a risk, and which I knew to potentially expect to go bad, and if I’m perfectly honest, I might’ve taken it anyway without the LLM”.
Without any indication as to how your situation maps to any of these (or more), the warning is, functionally, not particularly useful.
"I flipped a coin and the LLM called heads. I should have gone with tails..."
My guideline now for interacting with LLM is only to believe the result if it is factual and easily testable, or if I'm a domain expert. Anything else especially if I'm in complete ignorance about the subject is to approach with a high degree of suspicion that I can be led astray by its sycophancy.
Where are you often asking this question/getting these weak responses?
That is for certain types of therapy/clinical care. It is not always - and often isn’t - the case. Plenty of diagnoses and care protocols are not a matter of opinion or based on “you feeling there’s an issue” or deciding on your own there is no longer an issue.
Can anyone describe how to determine how a (professional, human) therapist is "a reliable agent" to make such a determination?
Meaning neither the LLM or the licensed therapist will voluntarily say, you are healed, you don't need me anymore.
Using LLMs for therapy is so deeply dystopian and disgusting, people need human empathy for therapy. LLMs do not emit empathy.
Complete disaster waiting to happen for that individual.
It is a first principle though so it helps to “stir the context windows pot” by having it pull in research and other shit on the web that will help ground it and not just tell you exactly what you prompt it to say.
One of the great myths of models in countless fields/industries. LLM’s are absolutely in no way objective.
Now if you want to say it’s an “outside opinion“ that’s valid. But do not kid yourself into thinking it is somehow empirical or objective
But it's better than talking to yourself or an abuser!
Sometimes people indeed just need validation and it helps them a lot, in that case LLMs can work. Alternatively, I assume some people just put the whole situation into words and that alone helps.
But if someone needs something else, they can be straight up dangerous.
They have world knowledge and are capable of explaining things and doing web searches. That's enough to help. I mean, sometimes people just need answers to questions.
In one way it's potentially worse than talking to yourself. Some part of you might recognize that you need to talk to someone other than yourself; an LLM might make you feel like you've done that, while reinforcing whatever you think rather than breaking you out of patterns.
Also, LLMs can have more resources and do some "creative" enabling of a person stuck in a loop, so if you are thinking dangerous things but lack the wherewithal to put them into action, an LLM could make you more dangerous (to yourself or to others).
An LLM generates non-deterministic information using sources you can’t even know, let alone evaluate, and is more primed to agree with you than give critical and objective evaluation. It is, at best, like asking your closest parent to help you through difficult interpersonal situations: The interaction is probably, subconsciously, going to be skewed enough towards soothing you that you just can’t consider it objective. The difference is that with an LLM, that’s deliberate. It’s designed in.
> Some of ELIZA's responses were so convincing that Weizenbaum and several others have anecdotes of users becoming emotionally attached to the program, occasionally forgetting that they were conversing with a computer. Weizenbaum's own secretary reportedly asked Weizenbaum to leave the room so that she and ELIZA could have a real conversation. Weizenbaum was surprised by this, later writing: "I had not realized ... that extremely short exposures to a relatively simple computer program could induce powerful delusional thinking in quite normal people."
I [half] jokingly tell people that a nice drive in a 911 [can be] better/cheaper therapy that going to a PhD.
Are LLMs mainly a tool for people with atrophying social skills who want all interaction to happen via a text prompt that always responds in the most soothing way? I can't think of another reason to replace human therapy with an LLM.
>"'Is it indeed?' laughed Gildor. 'Elves seldom give unguarded advice, for advice is a dangerous gift, even from the wise to the wise, and all courses may run ill...'"
This is the only way you should solicit personal advice from an LLM.
It’s even more maddening that this greedy maneuver was orchestrated based on LLM advice.
I’m glad the subnautica team won the lawsuit. Maybe I can play it now wothout feeling guilty
[0] - https://petergpt.github.io/bullshit-benchmark/viewer/index.v...
It _does_ love to explicitly agree with anything it finds in web search though.
(Anthropic tries to fight this by adding a hidden prompt that makes it disagree with you and tell you to go to bed, which doesn't help.)
I do like when opus is incredibly short in its responses to prompts that probably shouldnt have been made though. keeps me grounded a bit.
I try to focus on results. Things like an app that does what you want, data and reports that you need, or technical things like setting up a server, setting up a database, building a website, etc.
I have also found it useful for feedback and advice, but only once I have had it generate data that I can verify. For example, financial analysis or modelling, health advice (again factual based), tax modelling, etc, but again, all based on verifiable data/tables/charts.
I am very surprised on what Claude is capable of, across the entire tech stack: code, sysadmin, system integration, security. I find it scary. Not just speed, but also quality and the mental load is a difference of kind not quantity.
Personal advice on life decisions/relationships ? No way I would go there.
It is also good for me to know that the tools I have built, the data I have gathered, and my thinking approach places me as one of the most intelligent developers and analysts in the world.
I had to deal with a close family friend going through alcohol withdrawal and getting checked in at a recovery clinic for detox and used Claude heavily. The first thing I had it do as do that “deep research” around the topic of alcohol addiction, withdrawal, etc… and then made that a project document along with clear guidelines about how it shouldn’t make inferences beyond what it in its context and supporting docs. We also spent a whole session crafting a good set of instructions (making sure it was using Anthropics own guidelines for its model…)
Little differences in prompts make a huge deal in the output.
I dunno. It is possible to use these models for dumping crazy shit you are going through. But don’t kid yourself about their output and aggressively find ways to stomp out things it has no real way to authoritatively say.
(esp last sentence?)
Open two windows, ask it the same thing from starkly opposite perspectives, then see what it comes back with. If nothing else this exercise forces you to think deeply about what you're considering before you even see what the giant blob of matrix multiplication says about your situation.
Any LLM not sufficiently likable and helpful in the first two minutes was deleted or not further iterated on, or had so much retraining (sorry, "backpropagation") it's not the same as it started out.
So it's going to say whatever it "thinks" you want it to say, because that's how it was "raised".
The possibilities in "dangerous" fields are a bit more frightening. A general is much more likely to ask ChatGPT "Do you think this war is a good idea/should I drop a bomb", rather than an actually helpful tool - where you might ask "What are 5 hidden points on favor of/against bombing that one likely has missed".
The more you use AI as a strict tool that can be wrong, the safer. Unfortunately I'm not sure if that helps if the guy bombing your city (or even your president) is using AI poorly, and their decisions affect you.
Arguably, it already worked that way. The best way to climb the ranks of a 'dictatorial' organization (a repressive government or an average large business) is to always say yes. Adopt what the people from up above want you to use, say and think. Don't question anything. Find silver linings in their most deranged ideas to show your loyalty. The rich and powerful that occupy the top ranks of these structures often hate being challenged, even if it's irrational for their well-being. Whenever you see a country or a company making a massive mistake, you can often trace it to a consequence of this. Humans hate being challenged and the rich can insulate themselves even further from the real world.
What's worrying me is the opposite - that this power is more available now. Instead of requiring a team of people and an asset cushion that lets you act irrationally, now you just need to have a phone in your pocket. People get addicted to LLMs because they can provide endless, varied validation for just about anything. Even if someone is aware of their own biases, it's not a given that they'll always counteract the validation.
But sadly LLMs push all the right buttons that lead humans into that kind of behavior. And the marketing around LLMs works overtime to reinforce that behavior.
But instead if you ignore all that and use LLMs as a search tool, then you will get positive returns from using it.
Curious if you think a single person would have helped you make a better decision? Not everything works out. If a friend helped me make a decision I certainly wouldn’t blame them later if it didn’t work out. It’s ultimately my call.
It’s also possible to give bad advice wothout being malicious.
I have some friends who don’t give great life advice. We’re still friends, they’re just not the friends I go to when I need advice on a big decision.
It was extremely good at the other side too. You just have to ask. I can imagine most people don't try this, but LLMs literally just do what you ask them to. And they're extremely good and weighing both sides if that's what you specifically want.
So who's fault is it if you only ask for one side, or if the LLM is too sycophantic? I'm not sure it's the LLMs fault actually.
I wouldn't be so quick to discount the fact that you were essentially gaslit by an ass-kissing model that was RLHF'd into maximum persuasiveness. Models aren't just neutral tools, they're deliberately designed to be convincing.
Yes, your choices and actions are on you, but if a trillion dollar company gaslit you into thinking those were good choices to make, some of the responsibility is theirs, too.
This is like blaming yourself for an addiction to alcohol, junk food, gambling, or something else you have been relentlessly advertised to.
Sure, some of it falls on you, but there are corporations with infinite money spending most of it to manipulate your psyche into wanting the thing, trusting the thing, feeling empty without the thing.
I used to think obesity was self inflicted, for example. But then you notice how junk food companies are allowed to do whatever they want to get people hooked on their stuff. They can put up huge billboards, vending machines up a few metres from where you work, they even pump their smell out into the streets.
So let's not aim for a society where we blame victims of predatory marketing and carefully engineered addictive products. We all have weaknesses. Let's help each other out, not take advantage.
If you engage with the argument genuinely and steel man, you’ll see that is not what I said. I even emphasised it:
> some of it falls on you
You’re not a “helpless victim” but you’re also not fully to blame. Understanding that means understanding the problem and being more powerful to fix it. For example, if you’re addicted to TV or social media, you can make a concerted effort to improve your life by removing the problems at the external source (sell your TV, delete an account and app from your phone).
Another problem is believing you have a high emotional intelligence when there is no reliable way to quantify that - similar to "I believe I am very tall, but I don't know how tall I am and how tall are the others because there is no unit of measure for height", with the difference that for emotional intelligence there is no unit of measure and no correlation that can be established with anything to make at least an indirect measurement.
In that sense they can’t offer advice because the “know” nothing.
But they can reframe, they can reflect. They can take one idea and reflect it into another intellectual framework.
I’ve used them a lot like this to help get perspective on life decisions. But not for advice.
Try something like: I have to do x and y, give me multiple psych perspectives on this problem from different schools. I find this takes something abstract (your problem) and grounds it in things it actually knows (the sum of ingestible human knowledge).
Thankfully it was recoverable, but it really sobered me up on LLMs. The fault is on me, to be clear, as LLMs are just a tool. The issue is that lots of LLMs try to come across as interpersonal and friendly, which lulls users into a false sense of security. So I don't know what my trajectory would have been if I were a teenager with these powerful tools.
I do think that the LLMs have gotten much better at this, especially Claude, and will often push back on bad choices. But my opinion of LLMs has forever changed. I wonder how many other terrible choices people have made because these tools convinced them to make a bad decision.