News for hackers. Stuff that matters.
Re:Handbook.md shows that long policy documents do not relia... by cyanydeez on Wednesday July 29, 2026 @09:42AM (#49097427)
I believe the correct static instructions are about getting it at the right starting point for whatever class of projects you're working on; not as a continued referencable or "HOWTO" of what it's doing. They're all just "grooming" the LLM for future instructions.

The coding harness is what's getting it to continually align to your current instructions.

This is very obvious with local models.

Re:Handbook.md shows that long policy documents do not relia... by mcdeltat on Wednesday July 29, 2026 @09:48AM (#49097498)
Ok so what is the correct way to tell it "I don't care what is happening, you must uphold these rules at all times"? If it's not any configuration of .md files?
Re:Handbook.md shows that long policy documents do not relia... by pmarreck on Wednesday July 29, 2026 @10:06AM (#49097742)
You need to make the rule concrete somehow. I call it a "control". So for example, instead of instructing it "always run tests before committing", you (or you have it) make a git commit hook that always runs the tests first and that refuses the commit if they don't pass.

In this case, it is an advisory control only, because the LLM can also unhook that hook. And of course, it could also just disable the failing test(s) with some bullshit reason. But it is far better than assuming it will comply every time.

Then there is the "hard control", which is the inviolable that the LLM cannot bypass.

You need to move as much as is technically possible to either hard or (failing that) advisory controls.

Re:Handbook.md shows that long policy documents do not relia... by Muromec on Wednesday July 29, 2026 @10:45AM (#49098229)
>In this case, it is an advisory control only, because the LLM can also unhook that hook

you can in principle make a hook in the harness itself that will run an auditor prompt that checks it adhered to the policy, correct the model and also make it known in advance.

Re:Handbook.md shows that long policy documents do not relia... by dluxem on Wednesday July 29, 2026 @11:55AM (#49099151)
This appears to be what the claude code-review is doing:

https://github.com/anthropics/claude-code/blob/7ef6eec9d9ba8...

Re:Handbook.md shows that long policy documents do not relia... by pmarreck on Wednesday July 29, 2026 @06:36PM (#49103990)
this only checks whether the code is in compliance with CLAUDE.md though
Re:Handbook.md shows that long policy documents do not relia... by QuercusMax on Wednesday July 29, 2026 @11:53AM (#49099120)
I've seen pretty much every model including Claude bypass commit hooks constantly if there's even a tiny bit of friction, instead of pausing and asking me to help fix why it can't run the hooks (usually something that needs to be done outside the sandbox like `npm ci`).

That's why it's critical to have these checks run in a context that the model can't bypass, such as via github actions that block PR merges. Generally works pretty well for stuff at work (where we have all this stuff set up as part of continuous integration checks), but it means that for personal stuff I have to set up quite a bit more infrastructure to ensure that models don't just skip running tests (which they LOVE to do).

Re:Handbook.md shows that long policy documents do not relia... by pmarreck on Wednesday July 29, 2026 @06:37PM (#49103996)
Or just make it so that the commit hook can't be edited or bypassed by Claude?
Re:Handbook.md shows that long policy documents do not relia... by PunchyHamster on Wednesday July 29, 2026 @10:45AM (#49098231)
So you are essentially saying "you can't, you can only safeguard from effects of LLM eventually ignoring it"
Re:Handbook.md shows that long policy documents do not relia... by dbalatero on Wednesday July 29, 2026 @11:09AM (#49098548)
I think things like CI, linters, hooks, etc started as defense against humans eventually ignoring it (low blood sugar, under pressure, never cared in the first place, etc).

These seem like they would naturally extend to LLMs as well.

(I'm not pro-LLM on balance, but the solutions of today feel similar for humans and LLMs.)

Re:Handbook.md shows that long policy documents do not relia... by pmarreck on Wednesday July 29, 2026 @06:38PM (#49104001)
It is almost exactly like the controls from the auditing world, except as applied to AI's instead of people. In fact, having worked at Deloitte in the past is what led me to make the connection immediately
Re:Handbook.md shows that long policy documents do not relia... by pmarreck on Wednesday July 29, 2026 @05:16PM (#49103201)
Just like humans, yes.
Re:Handbook.md shows that long policy documents do not relia... by swatcoder on Wednesday July 29, 2026 @11:12AM (#49098590)
> you must uphold these rules at all times"?

You need to let go of the idea that this is something LLM's can do. They can't. At best, they can bias towards rules conformance with more or less likelihood, but coverage and conformance both go down super-linearly as you accumulate more rules, more context, and more output in a session. That's simply the nature of how these tools work and you need to engineer your workflows around it if you want to use them.

If you absolutely need some rules enforced, you need to adopt some framework for validating those rules that then rejects, reprocesses, or repeats any session that fails to satisfy them. In the best case scenario, this is some traditional deterministic validator (like a linter, compiler, analyzer, exhaustive test suite, etc in coding) but if you need to process in stochastic space because its something rich and ambiguous like natural language itself, then you want to dispatch a swarm very narrow, task-focused subagents that each validate against a very constrained subset. (And prepare yourself to have those to fail sometimes too. LLM's are noisy and cannot deliver strict rule enforcement on their own.)

Re:Handbook.md shows that long policy documents do not relia... by Muromec on Wednesday July 29, 2026 @10:05AM (#49097725)
Ask it to come back with a filled checklist and hand it over to a different agent with a fresh context window (three lines model). Or make it collapse the context and get back to the checklist.
Re:Handbook.md shows that long policy documents do not relia... by mwigdahl on Wednesday July 29, 2026 @09:50AM (#49097525)
Subagents whose only job is to review the actions of your other agents for rule compliance? It works reasonably well for me in complex workflows using Claude Code.
Re:Handbook.md shows that long policy documents do not relia... by carljungslabtek on Wednesday July 29, 2026 @10:01AM (#49097663)
Can I ask how you set this up? Like is there some way to have that run automatically, similar to “auto mode” for approvals, or do you have to invoke it regularly?
Re:Handbook.md shows that long policy documents do not relia... by mwigdahl on Wednesday July 29, 2026 @10:16AM (#49097872)
Sure! I use an orchestrator main agent whose only job is to run subagents through the workflow process I've defined. Part of that workflow is to invoke a review subagent at particular points to check the spec, the implementation plan, and the code for rule conformance.

The review subagent has its own definition and gets invoked with a specific target, so the context is very focused on just rule enforcement and I don't have problems with it skipping rules.

The whole workflow is packaged up as a plugin, but you don't need that to get this approach to work. It should be sufficient to have the rules you want enforced written out somewhere, and to either kick off a focused review agent manually referencing them, or do something like I did and have it be a defined part of your workflow (depending on the automation level you want).

Re:Handbook.md shows that long policy documents do not relia... by drooby on Wednesday July 29, 2026 @10:15AM (#49097849)
Hooks..

CI runs. Local Git hooks. Cursor also has hooks built into their agent. Other agent APIs probably have something similar.

Re:Handbook.md shows that long policy documents do not relia... by cyanydeez on Wednesday July 29, 2026 @10:34AM (#49098066)
in the plugins I'm using, it's basically _always_ adding information to the context. You're not going to do it manually; you can't go back in the context and add it because that'll break the cache. The way your programming harness works is by constantly reminding the LLM of the tools available.

A good programming harness is basically a stack. A good stack keeps building each layer. You _cannot_ pull things off the bottom of the stack because that's an expensive cache hit; but you can pull things off the top. So what your harness should be doing: <tools> <rules> <user content> onto each request _then_, when you get to the next request or result, pulling that out if there's some change.

So you can see it's the cache that's either exponentially growing or having to cache bust to keep it fresh.

Re:Handbook.md shows that long policy documents do not relia... by spIrr on Wednesday July 29, 2026 @09:45AM (#49097464)
As a hobbyist, I find it difficult to figure out how to make Claude stick with some repeating things I want it to do after every major action, like re-evaluate the completeness of tests, update the documentation, etc. And CLAUDE.md/AGENTS.md definitely did NOT help there, sadly.
Re:Handbook.md shows that long policy documents do not relia... by victorbjorklund on Wednesday July 29, 2026 @10:07AM (#49097754)
Hooks can be pretty useful for that. A hook when it is finished ”run tests suite and check coverage” ”check if your changes require updating the docs”
Re:Handbook.md shows that long policy documents do not relia... by Muromec on Wednesday July 29, 2026 @10:02AM (#49097675)
Don't use the default harness, write your own instead.
Re:Handbook.md shows that long policy documents do not relia... by TeMPOraL on Wednesday July 29, 2026 @10:14AM (#49097837)
Any way to do that AND use subscription instead of per-token pricing from SOTA providers?
Re:Handbook.md shows that long policy documents do not relia... by Muromec on Wednesday July 29, 2026 @10:30AM (#49098015)
If the harness runs on your localhost, but the inference doesn't, it usually means it's calling some API. Whether you want to break TOS of your provider like that or simply buy tokens from our friends from UTC+8 timezone is an open question.
Re:Handbook.md shows that long policy documents do not relia... by Supermancho on Wednesday July 29, 2026 @10:07AM (#49097756)
This is the way. Making your own agent to have a sticky memory context that is prepended to every execution is necessary to ensure each task is bounded by those precepts.
Re:Handbook.md shows that long policy documents do not relia... by Muromec on Wednesday July 29, 2026 @10:18AM (#49097883)
The trick I'm doing -- the model is given a tool that runs a prompt in the current thread to consolidate it's working memory and identity (it has a memory tool bound to the agent persona). When the prompt ends, the parts of memory that are marked as identity are merged together into a new system prompt, then the context restarts with only system prompt and this tool call surviving. Then it just keeps going.
Re:Handbook.md shows that long policy documents do not relia... by drob518 on Wednesday July 29, 2026 @10:42AM (#49098185)
Doesn’t that blow your token cache hit rate and balloon your costs (essentially everything is billed at the input token rate, not the cached rate)?
Re:Handbook.md shows that long policy documents do not relia... by haus20xx on Wednesday July 29, 2026 @11:29AM (#49098806)
If it moved a block like

<system_prompt>

<tools_etc>

<project_prompts>

<user_call>

<llm_response>

=>

<system_prompt>

<tools_etc>

//Removed project_prompts

<user_call>

<llm_response>

<user_call>

<project_prompts> //reinserted

<llm_response>

Cache would break but if you did instead

<system_prompt>

<tools_etc>

<project_prompts>

<user_call>

<llm_response>

<user_call>

<project_prompts> //Duplicated with new user messsage

<llm_response>

It wouldn't bust cache, it would just make your input prompts slightly larger. Technically inefficient as you're duplicating the same rules over and over but I imagine for a smallish checklist/principles that it is tremendously more efficient than a cache break every message

Re:Handbook.md shows that long policy documents do not relia... by cyanydeez on Wednesday July 29, 2026 @12:41PM (#49099784)
so first, we treat context like a stack. Things are popped onto the stack; if we try to remove anything but the top of the stack, we'll break the cache.

So a good harness should be popping tool prompts, user hints after every return from the top so the context of tools and user hints never repeats and are allways prefixed to the current user prompt.

I've never inspected any harness, but opencode with the dynamic context compaction plugin appears to do this well.

Re:Handbook.md shows that long policy documents do not relia... by Muromec on Wednesday July 29, 2026 @04:03PM (#49102295)
By the time it does 10 turns of tool calls it forgets about the nagging again. It can and does ignore direct instruction from the user prompt too, sometimes repeatedly
Re:Handbook.md shows that long policy documents do not relia... by cyanydeez on Wednesday July 29, 2026 @10:39AM (#49098136)
What I'm currently doing in a large refactor, is I created a super-run script; the super run script is devided into super-dev (Setup dev), super-test (run all tests), super-build (build artificats), super-e2e (test all artifacts), super-deploy (deploy finished).

Each super's sub functions should _fail hard_, and each script should be highly detailed; of course I'm not doing it myself, but in small increments of directed work, it can build up the necessary harness.

What I get is a CI that just starts with "run super-run.sh" and that gives it context, then each sub script provides context depending on if it succeeds or fails. If it fails, the agent is provided what it needs.

It's basically, you have to design the products of the AI to give itself the context. Another technique I'm testing out is a parallel set of files like <subject-module>.js, <subject-module>.test.js, <subject-module>.md which get pulled up if the Agent is looking for a file.