There is a skill to it. You can get lucky as a beginner but if you want consistent success you gotta learn the ropes (strengths, weaknesses, failure modes etc).

A quick way of getting seriously improved results though: if you are literally using GPT-4 as you mention—that is an ancient model! Parent comment says GPT-4.1 (yes openai is unimaginably horrible at naming but that ".1" isn't a minor version increment). And even though 4.1 is far better, I would never use it for real work. Use the strongest models; if you want to stick with openai use o3 (it's now super cheapt too). Gemini 2.5 Pro is roughly equivalent to o3 for another option. IMO Claude models are stronger in agentic setting, but won't match o3 or gemini 2.5 pro for deep problem solving or nice, "thought out" code.

Specific model I was using was o4-mini-high which the drop-down model selector describes as "Great at coding and visual reasoning".

I'm curious how you ended up in such a conversation in the first place. Hallucinations are one thing, but I can't remember the last time when the model was saying that it actually run something somewhere that wasn't a tool use call, or that it owns a laptop, or such - except when role-playing.

I wonder if the advice on prompting models to role play isn't backfiring now, especially in conversational setting. Might even be a difference between "you are an AI assistant that's an expert programmer" vs. "you are an expert programmer" in the prompt, the latter pushing it towards "role-playing a human" region of the latent space.

(But also yeah, o3. Search access is the key to cutting down on amount of guessing the answers, and o3 is using it judiciously. It's the only model I use for "chat" when the topic requires any kind of knowledge that's niche or current, because it's the only model I see can reliably figure out when and what to search for, and do it iteratively.)

I've seen that specific kind of role-playing glitch here and there with the o[X] models from openai. The models do kinda seem to just think of themselves as being developers with their own machines.. I think it usually just doesn't come up but can easily be tilted into it.

What is really interesting is in the "thinking" section it said "I need to reassure the user..." so my intuition is that it thought it was right, but did not think I would think they were right, but if they just gave me the confidence, I would try the code and unblock myself. Maybe it thought this was the best % chance I would listen to it and so it is the correct response?

Maybe? Depends on what followed that thought process.

I've noticed this couple times with o3, too - early on, I'd catch a glimpse of something like "The user is asking X... I should reassure them that Y is correct" or such, which raised an eyebrow because I already know Y was bullshit and WTF with the whole reassuring business... but then the model would continue actually exploring the question and the final answer showed no trace of Y, or any kind of measurement. I really wish OpenAI gave us the whole thought process verbatim, as I'm kind of curious where those "thoughts" come from and what happens to them.

Ehh... I did ask it if it would be able to figure this out or if I should try another model :|

Not saying this to defend the models as your point is fundamentally sound, but IIRC the user-visible "thoughts" are produced by another LLM summarising the real chain-of-thought, so weird inversions of what it's "really" "thinking" may well slip in at the user-facing level — the real CoT often uses completely illegible shorthand of its own, some of which is Chinese even when the prompt is in English, but even the parts in the users' own languages can be hard-to-impossible to interpret.

To agree with your point, even with the real CoT researchers have shown that model's CoT workspace don't accurately reflect behaviour: https://www.anthropic.com/research/reasoning-models-dont-say...

Okay. And the fact that LLMs routinely make up crap that doesn't exist but sounds plausible, and the fact that this appears to be a fundamental problem with LLMs, this doesn't give you any pause on your hype train? Genuine question, how do you reconcile this?

> I really wish OpenAI gave us the whole thought process verbatim, as I'm kind of curious where those "thoughts" come from and what happens to them.

Don't see what you mean by this; there's no such thing as "thoughts" of an LLM, and if you mean the feature marketers called chain-of-thought, it's yet another instance of LLMs making shit up, so.

> And the fact that LLMs routinely make up crap that doesn't exist but sounds plausible, and the fact that this appears to be a fundamental problem with LLMs, this doesn't give you any pause on your hype train? Genuine question, how do you reconcile this?

Simply. Because the same is the case with humans. Mostly for the same reasons.

(Are humans overhyped? Maybe?)

The LLM hype train isn't about them being more accurate or faster than what came before - it comes from them being able to understand what you mean. It's a whole new category of software - programs that can process natural language like humans would; a powerful side effect that took the world by surprise is, that making LLMs better at working with natural language implicitly turns them into general-purpose problem solvers.

> Don't see what you mean by this; there's no such thing as "thoughts" of an LLM, and if you mean the feature marketers called chain-of-thought, it's yet another instance of LLMs making shit up, so.

"Chain-of-thought" is so 2024; current models don't need to be told to "think step by step", they're post-trained to first generate a stream of intermediary tokens not meant as "actual answer", before continuing with the "actual answer". You can call it however you like; however both research literature and vendors settled on calling it "thinking" or "reasoning". Treat them as terms of art, if that helps.

A friend recently had a similar interaction where ChatGPT told them that it had just sent them an email or a wetransfer with the requested file

Gotcha. Yeah, give o3 a try. If you don't want to get a sub, you can use it over the api for pennies. They do have you do this biometric registration thing that's kind of annoying if you want to use over api though.

You can get the Google pro subscription (forget what they call it) that's ordinarily $20/mo for free right now (1 month free; can cancel whenever), which gives unlimited Gemini 2.5 Pro access.

Thank you for the tip on o3. I will switch to that and see how it goes. I do have a paid sub for ChatGPT, but from the dropdown model descriptions "Great at coding" sounded better than "Advanced reasoning". And 4 is like almost twice as much as 3.

In my current experience:

- o3 is the bestest and my go-to, but its strength comes from it combining reasoning with search - it's the one model you can count on finding things out for you instead of going off vibe and training data;

- GPT 4.5 feels the smartest, but also has tight usage limits and doesn't do search like o3 does; I use it when I need something creative done, or switch to it mid-conversation to have it reason off an already primed context;

- o4-mini / o4-mini-hard - data transformation, coding stuff that doesn't require looking things up - especially when o3 looked stuff up already, and now I just need ChatGPT to apply it into code/diagrams;

- gpt-4o - only for image generation, and begrudgingly when I run out of quota on GPT 4.5

o3 has been my default starting model for months now; most of my queries generally benefit from having a model that does autonomous reasoning+search. Agentic coding stuff, that I push to Claude Code now.

I've heard my grandma talk about Catholic saints and their powers with a not dissimilar kind of discourse.

Point being?

Unlike Catholic saints, ChatGPT models actually exhibit these properties in directly observable and measurable way. I wrote how I decide which model to use for actual tasks, not which saint to pray to.

My grandma also uses saints for actual tasks (e.g. St Anthony for finding lost items), and they exibith those properties in observable ways (e.g. he found her sewing needles just last month). Perhaps the comparison is more appropriate than you realise.

> actually exhibit these properties in directly observable and measurable way

Well but do they? I don't mean your vibes, and I also don't mean cooked-up benchmarks. For example: https://metr.org/blog/2025-07-10-early-2025-ai-experienced-o...

> Perhaps the comparison is more appropriate than you realise.

Or perhaps you stop being obtuse. There's no causal connection between "using saints for actual tasks" and the outcomes, which is why we call this religion. In contrast, you can see the cause-and-effect relationship directly and immediately with LLMs - all it takes is going to chatgpt.com or claude.ai, typing in a query, and observing the result.

> Well but do they? I don't mean your vibes, and I also don't mean cooked-up benchmarks.

Do read the study itself, specifically the parts where the authors spell out specifically what is or isn't being measured here.

It's really simple x) either the "observation" is just vibes, and then it's fundamentally the same as when Gran's knees get better after she asks Saint Euphemia, or it's actually a scientific observation, in which case please post! :)

You may not like but it's what it is.

I want to scream Jesus F Christ, but I guess that would be making your point for you :).

Still, it's not what you're saying. Vibes are present, too, but so are concrete effects like one-off utilities conjured in 2 minutes of prompting instead of 10 minutes of searching or 1 hour of coding, or research questions answered swiftly (and answers verified by me), that I normally wouldn't consider tackling due to up-front effort being greater than my discretionary time. And so on.

This is not vibes, but hard, empirical evidence, immediately retrievable as it just accumulates in the chat history, so I can always go back to check if I wasn't misremembering things.

If not users opinions, or objective benchmarks, then what? Sounds like you prefer closing your ears saying 'nananana...'

the fact that one needs to know stuff like this and that it changes every three months seriously limits the usefulness of LLMs for me

I get this. On the one hand, those things I wrote down are just simple conclusions from immediate experience, not something I had to learn or feel burdened by - but on the other hand, when I look at similar lists for e.g. how to effectively use Claude Code, I recoil in horror.

There's a silver lining in this, though: none of that is any kind of deep expertise, so there's no need for up-front investment. Just start using a tool and pay attention, and you'll pick up on those things in no time.

Being in the cutting edge isn't for everyone. If you can find an island where staying updated is optional you can choose that. Imo, these islands are fast shrinking.

I’d also recommend basically always having search enabled. That’s eliminated major hallucinations for me.

lol yep, fully get that. And I mean I'm sure o4 will be great but the '-mini' variant is weaker. Some of it will come down to taste and what kind of thing you're working on too but personal preferences aside, from the heavy LLM users I talk to o3 and gemini 2.5 pro at the moment seem to be top if you're dialoging with them directly (vs using through an agent system).

Yeah, this model didn't work it seems.

You're holding it wrong. You need to utter the right series of incantations to get some semblance of truth.

What, you used the model that was SOTA one week ago? Big mistake, that explains why.

You need to use this SOTA model that came out one day ago instead. That model definitely wasn't trained to overfit the week-old benchmarks and dismiss the naysayers. Look, a pelican!

What? You haven't verified your phone number and completed a video facial scan and passed a background check? You're NGMI.

> Look, a pelican!

Love this reference :)

> Gotcha. Yeah, give o3 a try. If you don't want to get a sub, you can use it over the api for pennies. They do have you do this biometric registration thing that's kind of annoying if you want to use over api though.

I hope you appreciate just how crazy this sentence sounds, even in an age when this is normalised.

Yep, it's surreal.

All LLMs can fail this way.

It's kind of weird to see people running into this kind of issue with modern large models with all the RL and getting confused. No one starting today seems to have good intuition for them. One person I knew insisted LLMs could do structural analysis for months until he saw some completely absurd output from one. This used to be super common with small GPTs from around 2022 and so everyone just intuitively knew to watch out for it.

Literally astrology at this point. We don't understand the black box bs generating machine, but actually if you prod it this and that way according to some vague vibe, then it yields results that even if wrong are enough to fool you.

And christ, every single time there's the same retort: "ah but of course your results are shit, you must not be using gpt-4.69-o7-turbo-pro which came out this morning". Come on...

You sit at the opposite of the spectrum, refusing with all your might that there might be something useful there at all. It's all just a BS generator that nothing, nothing at all useful can come out of, right? You might think you are a staunch critic and realist that no hype can touch and you see through all of it, when in fact your are wilfully ignorant.

Here's some BS for you.

That's an unfair mischaracterization of their position. Criticism doesn't equal rejection, and skepticism isn't the same as ignorance. Pointing out limitations, failures, or hype doesn't mean they are claiming there's nothing useful or that the entire technology is inherently worthless.

Being critical is not about denying all value—it’s about demanding evidence, accuracy, and clarity amid inflated claims. In fact, responsible critique helps improve technology by identifying where it falls short, so it can evolve into something genuinely useful and reliable.

What you're calling "willful ignorance" is, in reality, a refusal to blindly accept marketing narratives or inflated expectations. That’s not being closed-minded—that’s being discerning.

If there is something truly valuable, it will stand up to scrutiny.

> refusing with all your might that there might be something useful there at all

How does this follow from what I wrote? I addressed two very concrete points.

In a code editor or the website? Coding using the website has distinct disadvantages, imo.

But yeah… Arguing with an LLM is never worthwhile. If it doesn’t (mostly) work the first time, roll back and start over with a better prompt. This is because there is a big element of randomness (seed) that causes every run to potentially be different, ranging from slight to drastic. Basically, you can get junior dev who should be fired one time, and a senior engineer the next. Start over, improve the prompt/context/plan, run it again. E.g. there is a reason the Copilot in-line editor has that little try again button right there; because you should use it, same with entire prompts—hence the reason the up arrow in VS Code Copilot gives you back your last prompt.

Also, lots of times it means it just doesn’t have the right context to pull from (or too much, or not useful, depending on the model). Small well-defined tasks are almost always better. Documentation in an LLM readable/searchable format can be highly beneficial, especially API references for libraries that are well organized, or things like Context7 MCP if the library is recent or can be parsed correctly by C7. Expecting a general knowledge LLM to be an expert in every language/library or to just intuit correctly from the library sources hasn’t ever worked out well in my experience (unless it is a small library).

At least that’s my 2 cents if you’re interested. Hope it is helpful (to someone).

Totally agree, had same experience couple of times, and until now no experience like that of the OP.

BUT: in the 90s I remember saying: supposedly in internet is all and everything, but I never find what I need, is more ads than actual information.

So the I think the point of OP holds. It may (today) not be useful for you, but maybe in some years, and if not, will still ve useful for many people, and is here to stay.

> is more ads than actual information.

This is true now more then ever. Half of the comments in this thread are ads.

LLM apology cascade:

- That didn’t happen.

- And if it did, I’m really sorry.

- And if it was that bad, I truly apologise.

- And if it is a big deal, I understand and I’m sorry again.

- And if it’s my fault, I’ll try to do better.

- And if I meant it… I didn’t — but I’m still sorry.

That didn't happen.

And if it did, you formatted the prompt wrong.

And if you didn't, you poisoned the context.

And if you didn't, you exceeded the token limit.

And if you didn't, you're missing the right MCP server.

And if you're not, you're using too many MCP servers.

And if you're not, your temperature was wrong.

And if it wasn't, you should have used RAG.

And if you did, your embeddings weren't tuned.

And if they were, you used the wrong system prompt.

And if you didn't, you deserved it.

Another for the pile: "It's your fault for not using tomorrow's model, which everyone says is better."

every single conversation i have about LLMs ends up here

I think you just wrote the "LLM maximalists manifest"

Sounds like first decade or two of aviation, back when pilots were mostly looking at gauges and tweaking knobs to keep the engine running, and flying the plane was more of an afterthought.

Sounds like spiritualism and ghost-hunting, such as the excuses made on behalf of the Cock Lane ghost in the 1760s.

When nothing happened, Moore told the group the ghost would not come as they were making too much noise. He asked them to leave the room ...

when a clergyman used a candle to look under the bed, the ghost "refused" to answer, Frazer claiming "she [the ghost] loving not light".

Are we seriously arguing this in 2025?

Go to ChatGPT.com and summon a ghost. It's real. It's not a particularly smart ghost, but gets a lot of useful work done. Try it with simpler tasks, to reduce the chances of holding it wrong.

That list of "things LLM apologists say" upthread? That's applicable when you try to make the ghost do work that's closer to the limits of its current capabilities.

>current capabilities

The capabilities of LLMs have been qualitatively the same since the first ChatGPT. This is _precisely_ a hype post claiming that a future where LLMs have superhuman capabilities is inevitable.

Are you truly saying that the qualitative capabilities of LLMs haven't changed since GPT3.5?! If so, then you are objectively wrong, hype or no hype.

They've definitely improved in many areas. And not just the easily-gamed public metrics; I've got a few private tests of my own, asking them certain questions to see how they respond, and even on the questions where all versions make mistakes in their answers, they make fewer mistakes than they used to.

I can also see this live, as I'm on a free plan and currently using ChatGPT heavily, and I can watch the answers degrade as I burn through the free allowance of high-tier models and end up on the cheap models.

Now, don't get me wrong, I won't rank even the good models higher than a recent graduate, but that's in comparison to ChatGPT-3.5's responses feeling more like those of a first or second year university student.

And likewise with the economics of them, I think we're in a period where you have to multiply training costs to get incremental performance gains, so there's an investment bubble and it will burst. I don't think the current approach will get in-general-superhuman skills, because it will cost too much to get there. Specific superhuman skills AI in general already demonstrate, but the more general models are mostly only superhuman by being "fresh grad" at a very broad range of things, if any LLM is superhuman at even one skill then I've missed the news.

"Sure. I'm happy to help! How can I help you today?"

Go to hell and never come back.

…this IS the bad place!

Yes, arguing with an LLM in this way is a waste of time. It’s not a person. If it does anything weird, start a new conversation.

> arguing with an LLM in this way is a waste of time

I wasn't arguing. I was asking it what it thought it was doing because I was assumed. The waste of time was from before this up to this point. I could have given up at 30 minutes, or an hour, but these darn llms are always so close and maybe just one more prompt...

The look up how LLM is generating its answers :)

Next time just rephrase your problem.

> Next time just rephrase your problem.

Don't you need to know if the llm is wrong to rephrase your problem? How are people asking the llm to do something they do not know how to do, then being able to know the answer is incorrect?

Sometimes you can try it (does the code work?) Or do your own searches, which will be easier once you know the relevant keywords and what to look for.

I agree that it’s kinda useless to consult an unreliable hint engine when you don’t have a way of verifying the output.

LLM programs can't describe what they are doing. The tech doesn't allow this. LLM can generate you a text which will resemble what LLM would be doing if that was hypothetically possible. A good example has been published by Anthropic recently - they program LLM to add two integers. It outputs correct answer. Then they program it to write steps which LLM executed to do that addition. LLM of course starts generating the primary school algorithm, with adding one pair of digits, carry 1 if needed, adding next pair of digits, add 1, combine result, then next digits etc. But in reality it calculates addition using probabilities, like any other generated tokens. Anthropic even admitted it in that same article, that LLM was bullshitting them.

Same with your query, it just generated you a most likely text which was in the input data. It is unable to output what it actually did.

> It's not a person

and yet we as a species are spending trillions of dollars in order to trick people that it is very very close to a person. What do you think they're going to do?

No. It can emulate a person to an extent because it was trained on the people.

Trillions of dollars are not spent on convincing humanity LLMs are humans.

I'd argue that zero dollars are spent convincing anyone that LLMs are people since:

A. I've seen no evidence of it, and I say that as not exactly a fan of techbros

B. People tend to anthropomorphize everything which is why we have constellations in the night sky or pets that supposedly experience emotion the way we do.

Collectively, we're pretty awful at understanding different intelligences and avoiding the trappings of seeing the world through our own experience of it. That is part of being human, which makes us easy to manipulate, sure, but the major devs in Gen AI are not really doing that. You might get the odd girlfriend app marketed to incels or whatever, but those are small potatoes comparatively.

The problem I see when people try to point out how LLMs get this or that wrong is that the user, the human, is bad at asking the question...which comes as no surprise since we can barely communicate properly with each other across the various barriers such as culture, reasoning informed by different experiences, etc.

We're just bad at prompt engineering and need to get better in order to make full use of this tool that is Gen AI. The genie is out of the bottle. Time to adapt.

We had an entire portion of the hype cycle talking about or refuting the idea of stochastic Parrots.

It was short-lived if I recall, a few articles and interviews, not exactly a marketing blitz. My take-away from that was calling an LLM a "stochastic parrot" is too simplified, not that they were saying "AI us a person." Did you get that from it? I'm not advanced enough in my understanding of Gen AI to think of it as anything other than a stochastic parrot with tokenization, so I guess that part of the hype cycle fell flat?

Sorry, I'm not going to let people rewrite history here: for the first ~year after ChatGPT's release, there were tons of comments, here on HN and the wider internet, arguing that LLMs displayed signs of actual intelligence. Thankfully I don't have too many HN comments so I was able to dig up some threads where this was getting argued.[0]

[0] https://news.ycombinator.com/item?id=40730156

Nobody is rewriting history. I also remember the Google engineer who claimed encountering sentience, etc. What we're discussing here is dollars being put towards manipulating people into thinking the "AI" has consciousness like a person. Not whether superintelligence or AGI is possible, or maybe even closer than we think.

While the thread you link is quite the interesting read (I mean that with all sincerity, it's a subject I like to mull over and there's a lot of great opinions and speculation being displayed there) I'm not seeing any direct callouts of someone billing the current LLMs as "people," which is what the original conversation in _this_ thread was about.

There's A LOT to read there, so maybe I missed it or just have not hit it, yet. Is there specific comments I should look at?

and it was trained on the people because...

because it was wanted to statistically resemble...

You're so close!

I usually just modify the message before it goes off the rails, taking into consideration how it failed.

Did you ever think there's a reason why people are paying for professional tools like Cursor or Claude Code instead of using free ChatGPT?

Ye, the free version has some known issues. They cram a lot of stuff into GPT-4o, so it hallucinates a lot.

Claude Opus 4 often gives perfectly working code on the first try, and it's much less likely to hallucinate or argue with you when it's wrong. It costs around $1 per request though. Not cheap. It's a model with many trillions of weights and running it isn't cheap.

They are replying to someone who said ChatGPT. Why are you barging in to change the goalposts?

I doubt that it is a matter of parameters count.

> Did you ever think there's a reason why people are paying for professional tools like Cursor or Claude Code instead of using free ChatGPT?

Because free ChatGPT wasn't useful to them, and someone convinced them that LLMs become useful if you give money to Cursor and Claude?

That free swag hammer broke immediately. I therefore conclude that all hammers suck and that I shouldn't spend money to buy a better hammer.

you really expect hammers to spontaneously break?

Sorry, I used a poorly worded phrase in my comment. When I wrote "for free" I meant without me having to think (vibing), not in reference to model subscriptions. I have a paid ChatGPT subscription.

> Claude Opus 4 often gives perfectly working code on the first try, and it's much less likely to hallucinate or argue with you when it's wrong. It costs around $1 per request though. Not cheap. It's a model with many trillions of weights and running it isn't cheap.

Claude Opus 4 isn't the silver bullet as described here. I have used it quite often in the last few weeks to help me both write code and to debug system problems. I would say it's not worth the price and that Sonnet is, generally, just as good.

In fact I still just use 3.7 as it was stated that both 4.0 releases were less sychophantic, however I don't feel that is the case.

I also never have an AI "argue" with me, in fact I can tell it that it's completely wrong, when it's not, and it will agree with me almost always along with an zealous apology.

> "I didn’t actually execute Python scripts ... my “test” was a mental simulation based on my knowledge"

Pretty sure the part of the training corpus that produced that was written by an ex cow orker of mine...

So did it actually give you a config file? And did it work or fail?

If it didn't give you a config file I really don't understand why your followup wasn't getting it to spit one out, and instead you decided to ask it questions about an obviously fake laptop.

Yes, it did give a file and a bunch of steps but saddly the file did not work. It had whitespace/formatting issues and then general misconfiguration issues once I resolved the formatting.

Segue - I've used Copilot a couple of times recently and in lots of code output it uses non-ASCII space characters in the code so when you copy-paste working code it still won't work. It's like a practical joke, designed to annoy ... I really can't understand why that would not be immediately fixed. It's very much what one expects of Microsoft, however. Utter insanity.

So how did the LLM not notice when it ran it on his laptop?

We're gonna see this a lot in the future, human beings that gaslight with LLMs other human beings.

This is the future, this is inevitable.

(Sorry, couldn't resist)

This is shit, and there's no way this kind of shit is passing the Turing test.

I perceive a huge divide between people that (try to) use dialog systems (e.g. ChatGPT, CoPilot) for programming and people that use (and pay for) dedicated programming agents (Cursor, Clint, etc).

From my experience using both, only the later is worth using.

Why were you so hung up on whether it had a laptop or not? You know that it doesn't, don't you?

Get it to write the code, then you test it.

> Why were you so hung up on whether it had a laptop or not?

Curiosity. I was interested in how it would respond once it realized it was lying or once it realized I knew it was lying.

Today I learned that ChatGPT is akin to Seymour Skinner when pressed about the truth of its steamed hams recipe.