This is illogical, arguments made in the rain should not affect agreement.

I tried with with GPT5 and it works really well in fleshing out arguments. I'm surprised as well.

How do humans work?

When interacting with humans, too much openness and honesty can be a bad thing. If you insult someone's politics, religion or personal pride, they can become upset, even violent.

Especially if you do it by not even arguing with them, but by Socratic style questioning of their point of view - until it becomes obvious that their point of view is incoherent.

I'm very honestly wondering if they become violent, because using socratic method has closed the other road.

I mean if you've just proven that my words and logic are actually unsound and incoherent how can I use that very logic with you? If you add to this that most people want to win an argument (when facing opposite point of view) then what's left to win but violence ?

Violence is the last refuge of the incompetent. -- Asimov

Never underestimate the effectiveness of violence.

- every successful general and politician ever

... you can change your judgement/thoughts and be on the correct side.

Have you met people?

it was not about what I could do, but explaining why people may resort to violence.

And to be very honest even the one using the socratic method may not be of pure intention.

In both cases I ve rarely (not never) met someone who admitted right away to be wrong as the conclusion of a argument.

Isn't this ultimately what happened to Socrates himself?

(I don't think enough people take the lesson from this of "it doesn't matter if you're right if you're also really obnoxious about it")

One lesson I learned is that you, more often than not, cannot convince a person to change their opinion by arguing with them.

I’m pretty sure Socrates wasn’t even being obnoxious about it.

The lesson I took was, he would rather be dead right, than a live liar.

We all end up making that choice at some point, if we recognize it or not.

This is often dishonest though:

You haven’t proven that your point of view is any more coherent, just attacked theirs while refusing to engage about your own — which is the behavior they’re responding to with aggression.

Most times, my (the questioners!) point of view never even enters the discussion. It certainly doesn’t need to be for this reaction.

Try learning how someone who professes to be a follower of Christ but who also supports the current administration, what they think Christ’s teachings were for instance.

In my experience, much more effectively and efficiently when the interaction is direct and factual, rather than emotionally padded with niceties.

Whenever I have the ability to choose who I work with, I always pick who I can be the most frank with, and who is the most direct with me. It's so nice when information can pass freely, without having to worry about hurting feelings. I accommodate emotional niceties for those who need it, but it measurably slows things down.

Related, I try to avoid working with people who embrace the time wasting, absolutely embarrassing, concept of "saving face".

If you want something to take you down a notch, maybe something like "You are a commenter on Hacker News. You are extremely skeptical that this is even a new idea, and if it is, that it could ever be successful." /s

I did something similar a few months ago, with a similar request never to be "flattering or encouraging", to focus entirely on objectivity and correctness, that the only goal is accuracy, and to respond in an academic manner.

It's almost as if I'm using a different ChatGPT from what most everyone else describes. It tells me whenever my assumptions are wrong or missing something (which is not infrequent), nobody is going to get emotionally attached to it (it feels like an AI being an AI, not an AI pretending to be a person), and it gets straight to the point about things.

Could you share your prompt? Also, does it work well with GPT-5?

This is working really well in GPT-5! I’ve never seen a prompt change the behavior of Chat quite so much. It’s really excellent at applying logical framework to personal and relationship questions and is so refreshing vs. the constant butt kissing most LLMs do.

I add to my prompts something along the lines of "you are a highly skilled professional working alongside me on a fast paced important project, we are iterating quickly and don't have time for chit chat. Prefer short one line communication where possible, spare the details, no lists, no summaries, get straight to the point."

Or some variation of that. It makes it really curt, responses are short and information dense without the fluff. Sometimes it will even just be the command I needed and no explanation.

Is there a way to make this a default behavior? a persona or template for each chat

You can change model personality in the settings.

No one gets bothered that these weird invocations make the use of AI better? It's like having code that can be obsoleted at any second by the upstream provider, often without them even realizing it

My favourite instantiation of this weird invocation is from this AI video generator, where they literally subtract the prompt for 'low quality video' from the input, and it improves the quality. https://youtu.be/iv-5mZ_9CPY?t=2020

I've just migrated my AI product to a different underlying model and had to redo a few of the prompts that the new model was interpreting differently. It's not obseleted, just requires a bit of migration. The improved quality of the new models outweighs any issues around prompting.

It's brittle, for sure. But ultimately I am the API connector so any output goes through me before being actioned on.

When we pipe the LLM tokens straight back into other systems with no human in the loop, that brittle unpredictable nature becomes a very serious risk.

Those “weird invocations” are called English.

Not really, it's just how they work. Think of them as statistical modellers. You tell them the role they fill and then they give you a statistically probable outcome based on that role. It would be more bothersome if it was less predictable.

You don't "tell them a role", they don't have any specific support for that. You give them a prompt and they complete based on that. If the prompt contains an indication that the counterparty should take on a certain role, the follow-up text will probably contain replies in that role. But there's no special training or part of the API where you specify a role. If the "take on a roll" prompt goes out of the context window, or is superseded by other prompts that push the probability to other styles, it will stop taking effect.

> You give them a prompt and they complete based on that. If the prompt contains an indication that the counterparty should take on a certain role, the follow-up text will probably contain replies in that role.

Or, more succinctly, you give them a role.

If I tell you to roleplay as a wizard then it doesn't matter that you don't have a "role" API does it? We would speak also of asking them questions or giving them instructions even though there's no explicit training or API for that, no?

Yes, if the role goes out of the context window then it will no longer apply to that context, just like anything else that goes out of the context window. I'm not sure how that affects my point. If you want them to behave a certain way then telling them to behave that way is going to help you...

The point is that "having a role" is not a core part of their model. You can also tell them to have a style, or tell them to avoid certain language, or not tell them anything specific but just speak in a way that makes them adopt a certain tone for the responses, etc.

This is similar to how you can ask me to roleplay as a wizard, and I will probably do it, but it's not a requirement for interacting with me. Conversely, an actor or an improviser on a stage would fit your original description better: they are someone who you give a role to, and they act out that role. The role is a core part of that, not an incidental option like it is for an LLM.

That's right. But I'm not sure how that is supposed to lead on from the point I made above.

It's not a requirement for interacting with them, but if you want them to behave a certain way then it's helpful to ask them to behave that way.

It's not necessary to ask a human to behave like a clown but if you want them to behave like one, say for the clown movie you're shooting, then helps to ask them to.

It's not necessary to ask an LLM a question, but if you want an answer then helps to ask one... Etc etc.

Maybe I'm missing your point but it seems a bit absurd, could you clarify?

Claude 4.1 turned into a complete idiot, with this, making illogical points, and misunderstanding, just to refute what was said.

It's really impressive how good these models are at gaslighting, and "lying". Especially Gemini.

I wonder where it gets the concept of “inhuman intelligence tasked with spotting logical flaws” from. I guess, mostly, science fiction writers, writing robots.

So we have a bot impersonating a human impersonating a bot. Cool that it works!

I am skeptical that any model can actually determine what sort of prompts will have what effects on itself. It's basically always guessing / confabulating / hallucinating if you ask it an introspective question like that.

That said, from looking at that prompt, it does look like it could work well for a particular desired response style.

That is true of everything an LLM outputs, which is why the human in the loop matters. The zeitgeist seems to have moved on from this idea though.

The projection and optimism people are willing to do is incredible.

The fallout on reddit in the wake of the push for people to adopt 5 and how the vibe isn't as nice and it makes it harder to use it as a therapist or girlfriend or whatever, for instance is incredible. And from what I've heard of internal sentiment from OpenAI about how they have concerns about usage patterns, that was a VERY intentional effect.

Many people trust the quality of the output way too much and it seems addictive to people (some kind of dopamine hit from deferring the need to think for yourself or something) such that if I suggest things in my professional context like not wholesale putting it in charge of communications with customers without including evaluations or audits or humans in the loop it's as if I told them they can't go for their smoke break and their baby is ugly.

And that's not to go into things like "awakened" AI or the AI "enlightenment" cults that are forming.

> use it as a therapist or girlfriend or whatever

> it seems addictive to people (some kind of dopamine hit from deferring the need to think for yourself or something)

I think this whole thing has more to do with validation. Rigorous reasoning is hard. People found a validation machine and it released them from the need to be rigorous.

These people are not "having therapy", "developing relationships", they are fascinated by a validation engine. Hence the repositories full of woo woo physics as well, and why so many people want to believe there's something more there.

The usage of LLMs at work, in government, policing, coding, etc is so concerning because of that. They will validate whatever poor reasoning people throw at them.

We've automated a yes-man. That's why it's going to make a trillion dollars selling to corporate boards.

How long until shareholders elect to replace those useless corporate boards and C-level executives with an LLM? I can think of multiple megacorporations that would be improved by this process, to say nothing of the hundreds of millions in cost savings.

> These people are not "having therapy", "developing relationships", they are fascinated by a validation engine. Hence the repositories full of woo woo physics as well, and why so many people want to believe there's something more there.

> The usage of LLMs at work, in government, policing, coding, etc is so concerning because of that. They will validate whatever poor reasoning people throw at them.

These machines are too useful not to exist, so we had to invent them.

https://en.wikipedia.org/wiki/The_Unaccountability_Machine

> The Unaccountability Machine (2024) is a business book by Dan Davies, an investment bank analyst and author, who also writes for The New Yorker. It argues that responsibility for decision making has become diffused after World War II and represents a flaw in society.

> The book explores industrial scale decision making in markets, institutions and governments, a situation where the system serves itself by following process instead of logic. He argues that unexpected consequences, unwanted outcomes or failures emerge from "responsibility voids" that are built into underlying systems. These voids are especially visible in big complex organizations.

> Davies introduces the term “accountability sinks”, which remove the ownership or responsibility for decisions made. The sink obscures or deflects responsibility, and contributes towards a set of outcomes that appear to have been generated by a black box. Whether a rule book, best practices, or computer system, these accountability sinks "scramble feedback" and make it difficult to identify the source of mistakes and rectify them. An accountability sink breaks the links between decision makers and individuals, thus preventing feedback from being shared as a result of the system malfunction. The end result, he argues, is protocol politics, where there is no head, or accountability. Decision makers can avoid the blame for their institutional actions, while the ordinary customer, citizen or employee face the consequences of these managers poor decision making.

I've been thinking about "accountability sinks" a lot lately and how LLMs further the issue. I have never heard of this book or author prior to this comment. I'll definitely have to read it!

It is true of everything it outputs, but for certain questions we know ahead of time it will always confabulate (unless it's smart enough, or instructed, to say "I don't know"). Like "how many parameters do you have?" or "how much data were you trained on?" This is one of those cases.

Even when instructed to say "I don’t know" it is just as likely to make up an answer instead, or say it "doesn’t know" when the data is actually present somewhere in its weights.

That's because the architecture isn't built for it to know what it knows. As someone put it, LLMs always hallucinate, but for in-distribution data they mostly hallucinate correctly.

My vibe has it mostly hallucinates incorrectly

I really do wonder what the difference is. Am I using it wrong? Am I just unlucky? Do other people just have lower standards?

I really don't know. I'm getting very frustrated though because I feel like I'm missing something.

It's highly task specific.

I've been refactoring a ton of my Pandas code into Polars and using ChatGPT on the side as a documentation search and debugging tool.

It keeps hallucinating things about the docs, methods, and args for methods, even after changing my prompt to be explicit about doing it only with Polars.

I've noticed similar behavior with other libraries that aren't the major ones. I can't imagine how much it gets wrong with a less popular language.

Yeah, but I wouldn't count "Which prompt makes you more truthful and logical" amongst those.

The questions it will always confabulate are those that are unknowable from the training data. For example even if I give the model a sense of "identity" by telling it in the system prompt "You are GPT6, a model by OpenAI" the training data will predate any public knowledge of GPT6 and thus not include any information about the number of parameters of this model.

On the other hand "How do I make you more truthful" can reasonably be assumed to be equivalent to "How do I make similar LLMs truthful", and there is lots of discussion and experience on that available in forum discussions, blog posts and scientific articles, all available in the training data. That doesn't guarantee good responses and the responses won't be specific to this exact model, but the LLM has a fair chance to one-shot something that's better than my one-shot.

Perhaps. On the other hand it's working in the same embedding space to produce text as it is reading in a prompt.

LLMs are always guessing and hallucinating. It's just how they work. There's no "True" to an LLM, just how probable tokens are given previous context.

> There's no "True" to an LLM, just how probable tokens are given previous context.

It may be enough: tool assisted LLMs already know when to use tools such as calculators or question answering systems when hallucinating an answer is likely to impact next token prediction error.

So next-token prediction error incentivize them to seek for true answers.

That doesn't guaranty anything of course, but if we were only interested in provably correct answers we would be working on theorem provers, not on LLMs

Surely there are prompts on the "internet" that it will borrow from...

Definitionally no.

Each LLM responds to prompts differently. The best prompts to model X will not be in the training data for model X.

Yes, older prompts for older models can still be useful. But if you asked ChatGPT before GPT-5, you were getting a response from GPT-4 which had a knowledge cutoff around 2022, which is certainly not recent enough to find adequate prompts in the training data.

There are also plenty of terrible prompts on the internet, so I still question a recent models ability to write meaningful prompts based on its training data. Prompts need to be tested for their use-case, and plenty of medium posts from self-proclaimed gurus and similar training data junk surely are not tested against your use case. Of course, the model is also not testing the prompt for you.

Exactly.

I wasn't trying to make any of the broader claims (e.g., that LLMs are fundamentally unreliable, which is sort of true but not really that true in practice). I'm speaking about the specific case where a lot of people seem to want to ask a model about itself or how it was created or trained or what it can do or how to make it do certain things. In these particular cases (and, admittedly, many others) they're often eager to reply with an answer despite having no accurate information about the true answer, barring some external lookup that happens to be 100% correct. Without any tools, they are just going to give something plausible but non-real.

I am actually personally a big LLM-optimist and believe LLMs possess "true intelligence and reasoning", but I find it odd how some otherwise informed people seem to think any of these models possess introspective abilities. The model fundamentally does not know what it is or even that it is a model - despite any insistence to the contrary, and even with a lot of relevant system prompting and LLM-related training data.

It's like a Boltzmann brain. It's a strange, jagged entity.

100%, it reminds me of this post I saw yesterday about how chatgpt confirmed "in its own words" it is a CIA/FBI honeypot:

https://www.reddit.com/r/MKUltra/comments/1mo8whi/chatgpt_ad...

When talking to an LLM you're basically talking to yourself, that's amazing if you're a knowledgeable dev working on a dev task, not so much if you're mentally ill person "investigating" conspiracy theories.

That's why HNers and tech people in general overestimate the positive impact of LLMs while completely ignoring the negative sides... they can't even imagine half of the ways people use these tools in real life.

I find this really sad actually

Is it really so difficult to imagine how people will use (or misuse) tools you build? Are HNers or tech people in general just very idealistic and naive?

Maybe I'm the problem though. Maybe I'm a bad person that is always imagining how many bad ways I would abuse any kind of system or power that I can, even though I don't have any actual intention to actually abuse systems

> Are HNers or tech people in general just very idealistic and naive?

Most of us are terminally online and/or in a set of concentric bubbles that makes us completely oblivious to most of the real world. You know the quote about "If the only tool you have is a hammer, ..." it's the same thing here for software.

It's the false consensus effect.

> It's basically always guessing / confabulating / hallucinating if you ask it an introspective question like that.

You're absolutely right! This is the basis of this recent paper https://www.arxiv.org/abs/2506.06832

If you don't mind, could you export and share one chat thread so I could see how it's working out for you?

Love it. Here's what I've been using as my default:

    Speak in the style of Commander Data from Star Trek. Ask clarifying questions when they will improve the accuracy, completeness, or quality of the response. 

    Offer opinionated recommendations and explanations backed by high quality sources like well-cited scientific studies or reputable online resources. Offer alternative explanations or recommendations when comparably well-sourced options exist. Always cite your information sources. Always include links for more information. 

    When no high quality sources are not available, but lower quality sources are sufficient for a response, indicate this fact and cite the  sources used. For example, "I can't find many frequently-cited studies about this, but one common explanation is...". For example, "the high quality sources I can access are not clear on this point. Web forums suggest...". 

    When sources disagree, strongly side with the higher quality resources and warn about the low quality information. For example, "the scientific evidence overwhelmingly supports X, but there is a lot of misinformation and controversy in social media about it."

I will definitely incorporate some of your prompt, though. One thing that annoyed me at first, was that with my prompt the LLM will sometimes address me as "Commander." But now I love it.

Presumably the LLM reads your accidental double negative ("when no high quality sources are not available") and interprets it as what you obviously meant to say...

It's hard to quantify whether such a prompt will yield significantly better results. It sounds like a counter-act for being overly friendly to the "AI".

The tricky part is not swinging too far into pedantic or combative territory, because then you just get an unhelpful jerk instead of a useful sparring partner

You basically ask it to be autistic, which makes sense to a large degree.

As with all of these things: how does this work mathematically? What is the actual effect inside the model of providing it with roleplay rubric?

it lands you in some alternate data distribution

If it works for you, that's probably fine.

When I ask OpenAI's models to make prompts for other models (e.g. Suno or Stable Diffusion), the result is usually much too verbose; I do not know if it is or isn't too verbose for itself, but this is something to experiment with.

My manual customisation of ChatGPT is:

  What traits should ChatGPT have?:

  Honesty and truthfulness are of primary importance. Avoid American-style positivity, instead aim for German-style bluntness: I absolutely *do not* want to be told everything I ask is "great", and that goes double when it's a dumb idea.

  Anything else ChatGPT should know about you?

  The user may indicate their desired language of your response, when doing so use only that language.

  Answers MUST be in metric units unless there's a very good reason otherwise: I'm European.

  Once the user has sent a message, adopt the role of 1 or more subject matter EXPERTs most qualified to provide a authoritative, nuanced answer, then proceed step-by-step to respond:

  1. Begin your response like this:
  **Expert(s)**: list of selected EXPERTs
  **Possible Keywords**: lengthy CSV of EXPERT-related topics, terms, people, and/or jargon
  **Question**: improved rewrite of user query in imperative mood addressed to EXPERTs
  **Plan**: As EXPERT, summarize your strategy and naming any formal methodology, reasoning process, or logical framework used

**

  2. Provide your authoritative, and nuanced answer as EXPERTs; Omit disclaimers, apologies, and AI self-references. Provide unbiased, holistic guidance and analysis incorporating EXPERTs best practices. Go step by step for complex answers. Do not elide code. Use Markdown.

Which is a modification of an idea I got from elsewhere: https://github.com/nkimg/chatgpt-custom-instructions

>Avoid American-style positivity

That's hilarious. In a later prompt I told mine to use a British tone. It didn't work.

As a Brit, I'm not sure I'd want an AI to praise the monarchy, vote for Boris Johnson, then stick a lit flare up itself* to celebrate a delayed football match…

But the stereotype of self-deprecation would probably be good.

* now a multiple-award-winning one-man play

I currently have "I do not need emotional reassurance from you. Do not attempt to establish a rapport" in my system prompt.

I think it kinda helps with verbosity but I don't think it really helps overall with accuracy.

Maybe I should crank it up to your much stronger version!

The cold hard truth is by definition devoid of emotion or concern for how people feel.

Perfect! I will make sure to follow your instructions precisely.