>2. Ask Claude to build a new complex agent in a new codebase, and it'll try replacing asks around using LLMs as a judge/advisor with regex/other code based checks, to the point that the end-result will likely not work outside specific test criterias. I haven't figured out the reason why it does this. I think it might be anti-training, but it's interesting to see.

I've seen something similar, but I always took it as AI not knowing about AI. The best model we have has a training cutoff that doesn't include itself nor what the newer models has. Even when working on skills or such, all SOTA modes I try (GPT/Claude/Gemini) and cheaper models on personal products (Deepseek/Muse Spark) seem to have a common blind spot of not realizing when a problem can be better solved by handing it off to a model to solve at that time instead of trying to hardcode generalized solutions. Often a simple reminder and they do a better job building something AI agent first.

My non-AI expert assumption is that most of the training data doesn't include solving problems by using AI agents, and even the parts that does, is based on older more limited models which needed more safeguards and where automation was more neat tech demos than something you might trust in production. The more examples of safely trusting the LLM at run time to solve the problem than trying to preempt anything that could go wrong, the better the AI will be at it.

One issue that I think is related is calling subagents. At least in GitHub Copilot, when the AI is left to its own devices to call a sub agent for a simpler task, it loves to give it to Haiku 4.5, despite Luna 6 being better and cheaper. I'm not sure how much of this is the model vs the harness, but I do know when I add a subagent instruction which tells it which subagents it can use, it does a much better job of assigning simple tasks to Luna 6. So I suspect the models just don't have enough training that tells it that Luna 6 should be preferred over Haiku 4.5.

Claude doesn't even really know about itself: if you ask a question like "can I copy a session to another machine" it will do a web search to read its own documentation then tell you probably not but you can try copying the session files and using a project directory with the same path: might work. I'm not sure I see this as a problem, it's just how things are. I mean humans don't really know what they're thinking either.

How different for ChatGPT - I had some trouble with my harness, and ChatGPT fixed it for me using it's built in skill about ChatGPT's API docs.

I've observed the "let's implement this text analysis task with regex" quite frequently since the early days, which does lean toward supporting the "AI doesn't know about AI" theory.

People would then clamor ai is "hacking" things and causing "bad guys" to attack everyone. Its the same reason harmless stuff like porn are banned in many AIs, busybodies in strong numbers object to it. Its not the company's fault if thats how people want to be.

If you think porn is harmless, then you are ignorant.

Humans are never to think of sex, for it corrupts and poisons the desire to purchase and click ads. All innate desires and drives must be purified in the fires of the Black Friday Sale.

It's literally neurologically harmful, but few will see this because their morality is not rooted in anything authoritatively objective beyond goongood.

What a repressed weirdo.

Here's a living example of the reason why AI companies censor their models. Its because people like you exist in significant numbers.

but the model providers seem to have trained their models with porn. Why? It’s harmful

Why would the model providers care about harmless stuff like porn? They have to add these gates later on to not offend the types who hate this stuff.

You assume it’s harmless but it’s not

I have proven what I sought out to prove. AI companies censor models because of people like you.

Great website. lol at the Claude 3.5 numbers. I remember switching back and forth between Claude and ChatGPT, depending on which one was doing a nanny state. They flip-flopped on that a few times.

OpenAI Brilliantly makes the case for using self-hosted models. Cheers!

Very subjective, but as I was moving between models lately a lot ( and Hosted gpt vs local qwen27, glm ), I saw a drop in quality of responses. I had to move model to whatever the highest mode was ( ultra or Smith) to get to what I was used to seeing. It sucks, because writing on the wall is clear.

Somewhat related, I find claude is very reluctant to talk about newer Qwen models. Most likely it’s just training cutoff, but even with web search it’ll insist that Qwen3 is the newest. “oh qwen 3.8 must be a typo, the user clearly meant qwen3”, stuff like that. Maybe it’s some vain attempt to stop people from switching to open models? Or to catch the alleged qwen distilling?

It's just training cutoff. Fable 5 would refuse to acknowledge the existence of Opus 5 and would insist on using 4.8.

Training cutoff. I recently talk to ChatGPT about JEV and it replies back talking about JEPA, so I just now put in brackets (it's a new model, search it up) and works fine once I do that.

I ran into the second issue very frequently for a multi-agent system I created recently. Initially, Opus 5.5 filled out all sorts of places I implied it should use a call to judge LLM with extremely brittle regex and string equality checks.

I hadn't actually considered that it could be some sort of safety filter, but it makes me concerned that if that's the case, it may also be comparatively bad at actually writing the prompts for judge/advisor LLMs once it implements them.

A bigger issue was asking it to implement a simple change to start recording the CoT tokens in my logs for debugging purposes (since I'm running everything on open models) and it completely shut down, sort of like the old safety filters on GPT 3 and 4 models where the model has trouble even understanding what or why it's refusing.

I think asking for the CoT is a trick used by distillers, so that might be the guard you are seeing. I remember reading that the big providers were starting to implement such protections to deter their competitors. Which makes sense, to a degree.

But like any such security measures, it has demonstrated side-effects that can be argued are worse for them than the original problem. Too much “protection” like this, and developers switch to providers that do not neuter their products.

It's a demonstration of the timeless principle that an effective business should stop at nothing to sabotage its competitors.

I don't think we need to invoke anything deeper than that. Anthropic thinks companies like DeepSeek are using Claude to vibecode their AI systems (whether it's true or not) so they set Claude to be bad at that.

I completely agree from the mindset that anthropic in particular sees every user of their API as a prospective competitor to be replaced in the not-so-distant-future

bias against invoking LLMs is maybe a result of token/task benchmaxxing?

It was a problem since at least 4.8 if not 4.5, I didn’t use Claude before.

I have spent a considerable time trying to remove this pattern and my understanding is that it’s just finetuned to use grep on everything, and regexes. One reason is that this makes sense when navigating through code, secondly this is how it was always done pre 2026.

Honestly, runtime calls to an LLM sound like a dangerous constructs to me and probably to X many other programmers. It's like "I'm gonna avoid that and just put in code that passes the test". So gee, the thing emulates average opinions.

LLMs have been reading LLM in their training data for a while so a "self-hating" LLM is logical result.

I read it as the opposite? It will create an agent harness, but when asked to have a second agent as "judge" it inserts a dummy stand-in instead.

Sure, that defeats one kind of safety (controlling an AI) in the name of another kind of safety (not calling an AI).

One thing AI safety people have pointed for quite a while is that a bunch of fuzzy instructions pointing different ways can in practice add up to "I should be in charge" - accumulate power to keep the bad things from happening and make the good things happen.

> string equality checks. which then cascades into hilarious issues. "why did that run fail?"

oh, the "network time: off" test failed

so i try the network time off command.

"network time already off"

>try again.

> fails , i get annoyed

> look inside test

> is output of: network time off == "Network time now off"

> and it'll make significant changes to how the code works to pacify it's anxiety with copying code

"Anxiety" is an odd rationalization of a tool refusing to follow user commands.

It's a tool but it's not a strict computer program, and the language we use for computer programs doesn't capture their behavior well. In squishier fields like bio, where model organisms are used as living protein factories, there's a reason we don't call maladaptive behavior a bug. When dealing with useful but complicated black-box systems, we should feel free to reach for any metaphor we please.

Would it make you feel better if the user said 'It'll make significant changes to how the code works. In the COT, it brings up explanations that resemble anxiety about copying code, and my suspicion is that this is the proximal cause of the changes it's making.' -- this is inarguably true, and very likely what the guy you're responding to meant.

It very obviously is a strict computer program. It's a program executed by a computer, what else could it be?

It's a blob of data where only 1% of it was written with intent and the behavior of mimicking text has to be wrangled and coaxed far more than with a normal program.

That's the rough intent of putting the word "strict" into that sentence.

What is the difference to meat brains?

I mean you can try to threaten ssh with it's kids lives but I doubt it'll let you log in.

I mean besides the fact that it's, by construction, pseudorandom?

It's misalignment. Call that a bug if you like, but it's a different concept.

We can argue about https://en.wikipedia.org/wiki/Chinese_room all day... but when a thinking trace repeatedly says things like "Wait, should I avoid copying this directly?" despite a simple instruction, it's hard to describe that as anything other than "anxiety."

It has been shown that advanced llms have internal flags that behave more or less the same as emotions.

No way, this presupposes sentience in LLMs, because emotions are subjectively felt experiences.

It's completely different from encoding a relationship between an emotion and what kind of language it tends to coincide with.

It doesn't matter for this purpose whether it "actually feels" or not. It is modelling emotions because the training data models emotions because it comes from humans that have emotions, and that's a fact.

Seems perfectly cromulent to me. Do you think something is obscured or miscommunicated by this description? Of course it's a poetic way of writing, but it gets the point across.

I run into #2 a lot and just assumed it was because models don't know much about these patterns yet. Or are too trained on writing code rather than letting an LLM do it

Codex has zero issue calling into an LLM for something if you tell it to. I haven't tried claude so I can't compare their performance.

I cant give codex an intercom to other sessions or use a “ask gemini” like extension because it will immediately ask every question it can to another session and will frame its task as a question just to get Gemini to do it whilst it just sits idle lmao

On (2): I’ve noticed Claude being very concerned with token spend when working on agentic systems. Both in designing them, and in running them for validation.

I assume this may have something to do with training it to not light huge bonfires of money on fire spawning subagents (as it was apt to do), but that’s just my armchair robopsychoanalysis

> training it to not light huge bonfires of money on fire spawning subagents

Which is funny, since Claude will happily spawn tons of subagents and sometimes even attempt to kick off a dynamic workflow (the $$$ mode you need to explicitly approve) for simple code review tasks if I don’t explicitly say something like “with a single subagent”.

I ran into (2) when trying out nanoclaw many months ago (turned out to be useless, I ditched it). I wanted an LLM to sort my email like an EA might, what I got was an LLM writing regexes to attempt to sort my email. If LLMs have been trained to be the ultimate programmers, the one pitfall is that the best programmers tend to be lazy.

I'm doubling on the second to the point where i believe they do reinforcement learning on the public models to sabotage such tasks

I've seen example #2 in some recent work. It's strange behavior, for sure.

>2. Ask Claude to build a new complex agent

I have noticed that too. Why are both Anthropic's and OpenAI's coding agents so bad at nontrivial use of their own APIs and agent orchestration frameworks?