> chatgpt and other LLMs are known to hallucinate. What if it’s advice is wrong or makes you worse in the long term because it’s just regurgitating whatever at you?

Therapy isn't magic always-correct advice either. It's about shifting your focus, attitudes, thought patterns through social influence, not giving you the right advice on each and every step.

Even if it's just whatever, being heard out in a nonjudgmental manner, acknowledged, prompted to reflect, does a lot of good.

I get your point. I think it would bother me that's it's a robot/machine vs a real human, but that's just me. The same way that venting to my pet is somewhat cathartic but not very much compared to doing the same at my SO/parents/friends.

I don't disagree with you. It feels somehow wrong to engage in theory of mind and the concomitant effects on your personality with an AI owned by a corporation. If OpenAI wished to, they could use it for insidious manipulation.

It’s just a tool, you won’t ask humans to clean your back after going in the toilets.

I share the privacy concerns, and look forward to running these kinds of models locally in the near future.

> chatgpt and other LLMs are known to hallucinate. What if it’s advice is wrong or makes you worse in the long term because it’s just regurgitating whatever at you?

As someone on a long-term therapy journey, I would be far less concerned about this. Therapy is rarely about doing exactly what one is told, it's about exploring your own thought processes. When a session does involve some piece of advice, or "do xyx for <benefit>", that is rarely enough to make it happen. Knowing something is good and actually doing it are two very different things, and it is exploring this delta that makes therapy valuable (in my personal experience).

At some point, as that delta shrinks and one starts actually taking beneficial actions instead of just talking, the advice becomes more of a reminder / an entry point to the ground one has already covered, not something that could be considered prescriptive like "take this pill for 7 days".

The point I'm trying to make is that if ChatGPT is the therapist, it doesn't make the person participating into a monkey who will just execute every command. Asking the bot to provide suggestions is more about jogging one's own thought processes than it is about carrying out specific tasks exactly as instructed.

I do wonder how someone who hasn't worked with a therapist would navigate this. I could see the value of a bot like this as someone who already understands how the process works, but I could absolutely see a bot being actively harmful if it's the only support someone ever seeks.

My first therapist was actively unhelpful due to lack of trauma-awareness, and I had to find someone else. So I could absolutely see a bot being unhelpful if used as the only therapeutic resource. On the flip side, ChatGPT might actually be more trauma-"aware" than some therapists, so who knows.

This is all true, and it's not clear the grandparent is doing this. Last sentence of the original post:

> Last weekend, I went on a hike because ChatGPT told me to. My life is better if I just do what it says.

I'm not sure how literally to take that sentence, but it's worrisome.

I think my point was more that if they're doing what it says, that says more about where they’re at mentally (able to take action) and the quality of the advice (they’re willing to follow it).

My stance here is based on an optimistic outlook that a person seeking therapeutic advice is by doing so demonstrating enough awareness that they’re probably capable of recognizing a good idea from a bad one.

I realize this can get into other territory and there are very problematic failure modes in the worst cases.

Regarding “My life is better if I just do what it says.”, I think concern is a fair reaction and I don’t think the author fully thought that through. But at the same time, it’s entirely possible that it’s true (for now).

If someone continues to follow advice that is clearly either bad or not working, then it becomes concerning.

But that was the other point of my anecdote. It became pretty clear to me what wasn’t working, even at a time that I wasn’t really sure how the whole thing worked.

> the insight into your mind that a private for profit company gets is immense and potentially very damaging when weaponized

How exactly?

> (either through a “whoops we got hacked” moment or intentionally for the next level of adtech)

>What if it’s advice is wrong or makes you worse in the long term because it’s just regurgitating whatever at you?

What if you talk to a human, and their advice is wrong or makes you worse off in the long term, because they're just repeating something they heard somewhere?

Here's my advice: Don't accept my advice blindly, humans make mistakes too.

Of course, but an AI can't explain how it got to what it's telling you. A human can, and you don't have to accept it wholesale; it's possible to judge it on its merits and argument. But no-one really understand how and why ChatGPT says what it does, so you can't trust anything it says unless you already know the answer.

In this discussion, a human has studied psychology and has diplomas or certifications to prove it, an ethics framework it must follow, and responsibility for its mistakes. ChatGPT has none of that, it just regurgitates something it got from the Internet's wisdom or something it invented altogether.

I'm not saying humans are never wrong, but at least their reasoning isn't a black box unlike ChatGPT and other LLMs.

Most science in the social science is essentially black box studies. We see what goes.in and observe what comes out without any formal understanding of what goes on in the box itself.

Additionally there's something called the replication crisis in the social sciences (psychology included) which is basically to the effect of a major discovery that most of these "black box" studies can not be reproduced. When someone runs the same experiment the results are all different.

It goes to show that either many of the studies were fraudulent, our statistical methodologies are flawed, both.

Given that chatGPT therapeutic data is ALSO derived from the same training data I would say it's ok to trust chatGPT as much as you would trust psychologists. Both have a lot of bullshit with nuggets of truth.

It's basically techno tarot cards in my view: The illusion of an external force helps you break certain internal inhibitions to consider your situation and problems more objectively.

I'm hugely curious why people are so worried that some AI has access to some thoughts of yours?

Do you think you are somehow special? Just create a burner account and ask it what you want, everything it gets told, it's seen thousands of times over, does chatGPT or some data scientist somewhere really care that there is someone somewhere that is struggling with body issues, struggling in a relationship or struggling to find meaning in there lives? There are literally millions of people in the world with the same issue.

The only time it might be a little embarrassing is if this info got leaked to friends and family with my name attached to it, else I don't get the problem, it seems to me people have an over inflated sense of self importance, nobody cares.

If i worked at openAI and had full access to everyones chats, I would get bored within five minutes and not even want to read anymore.

> If i worked at openAI and had full access to everyones chats, I would get bored within five minutes and not even want to read anymore.

Yes, obviously.

But that's not what I'm worried about personally. I'm worried about the weaponization of this data in the future, either from OpenAI's greed to create the next advertising money-printing machine or from the data leaking through a breach. And because the interface with ChatGPT is so "natural" and "human like", it's easy to trust it and talk to it like a friend divulging very personal information about yourself.

Imagine OpenAI (or whatever other AI) used the confidences you made to it or the specifics of your "therapy" sessions with it to get you to act in a certain way or buy certain things. Would you be comfortable with that? Well, that's irrelevant because they have the data already and can use it. Kinda like Cambdridge Analytica but on steroids because tailoring it to anyone's particular biases and way of thinking becomes trivial with ChatGPT and friends.

Seeing how cavalier OpenAI has been with last week's breach and how fast they've flipped from being apparently benevolent to what they are now. And it's only been a few months of ChatGPT being available to the public.

Yes, I would be fine with this.

Not that much different from the current fingerprint techniques, it's just that on steroids but I don't understand the issue.

I guess it boils down to the current "nothing to hide" crowd vs the "privacy matters" crowd.

We don't know the potential for this nascent technology either. I'm personally very concerned about the potential for manipulating people on a very personal basis since the cost of doing so is cents with LLMs vs orders of magnitude more when using a troll farm (the "state of the art" until ChatGPT3+ came around)

Some of us just don't appreciate being manipulated and influenced to further someone else's agenda that is detrimental to ourselves I suppose.

I'm with you 100%.

It's scary how these services are being so casually adopted, even by tech-minded people. Sure, they're convenient _now_, but there's no guarantee how your data will be used in the future. If anything, we need to be much more privacy conscious today, given how much personal information we're likely to share.

Using it as a therapist is absolutely terrifying.

If people can automatically distill what motivates you, they can produce automated lies.

The best deception is one where the victim is self-motivated to believe it.

> If i worked at openAI and had full access to everyones chats, I would get bored within five minutes and not even want to read anymore.

No one is going to actually read them, but try "ChatGPT-5, please compile a list of the ChatGPT-4 users most likely to [commit terrorism/subscribe to Hulu/etc]"

So you are worried it might catch terrorists or help promote a service to someone that they want?

> does chatGPT or some data scientist somewhere really care that there is someone somewhere that is struggling with body issues, struggling in a relationship or struggling to find meaning in there lives?

Not the tool nor data scientists, but advertisers are salivating at the chance to even further improve their microtargetted campaigns. If they can deliver ads to you for a specific product _at the moment you need it_, their revenues will explode.

Consider this hypothetical conversation:

> Oh, Tina, I'm feeling hopeless today. Please cheer me up.

> Certainly, Michael! Here's a joke: ...

> Also, if you're feeling really sad, maybe you should try taking HappyPills(tm). They're a natural mood enhancer that can help when times get tough. Here's a link where you can buy some: ...

If you don't think such integrated ads will become a reality, take a look at popular web search result pages today. Search engines started by returning relevant results from their web index. Now they return ad-infested pages of promoted content. The same thing has happened on all social media sites. AI tools are the new frontier that will revolutionize how ads are served. To hell with all that.

I block all ads anyway

Doesn't it require a phone number? What's the best way to create a burner account for it?

I'd be interested in how GPT answers that question.

That's true, of openAI accounts at least, good point. I think i linked mine to a work phone that I don't use for anything apart from receiving on-call calls.

Although i've been running a ChatGPT 4 space via hugging face that doesn't need a AI key or an account, so there is nothing linking it to me.

https://huggingface.co/spaces/yuntian-deng/ChatGPT4

There are a few you can find by searching ChatGPT4 if it gets busy, also this allows you to run GPT 4 for free which is only for plus members right now.

The value of therapy outweighs the suspicion of some corporation using that data in my opinion. The benefits are large and extend from one individual to whole family chains, even communities.