News for hackers. Stuff that matters.
Re:Handbook.md shows that long policy documents do not relia... by pmarreck on Wednesday July 29, 2026 @10:06AM (#49097742)
You need to make the rule concrete somehow. I call it a "control". So for example, instead of instructing it "always run tests before committing", you (or you have it) make a git commit hook that always runs the tests first and that refuses the commit if they don't pass.

In this case, it is an advisory control only, because the LLM can also unhook that hook. And of course, it could also just disable the failing test(s) with some bullshit reason. But it is far better than assuming it will comply every time.

Then there is the "hard control", which is the inviolable that the LLM cannot bypass.

You need to move as much as is technically possible to either hard or (failing that) advisory controls.

Re:Handbook.md shows that long policy documents do not relia... by Muromec on Wednesday July 29, 2026 @10:45AM (#49098229)
>In this case, it is an advisory control only, because the LLM can also unhook that hook

you can in principle make a hook in the harness itself that will run an auditor prompt that checks it adhered to the policy, correct the model and also make it known in advance.

Re:Handbook.md shows that long policy documents do not relia... by dluxem on Wednesday July 29, 2026 @11:55AM (#49099151)
This appears to be what the claude code-review is doing:

https://github.com/anthropics/claude-code/blob/7ef6eec9d9ba8...

Re:Handbook.md shows that long policy documents do not relia... by pmarreck on Wednesday July 29, 2026 @06:36PM (#49103990)
this only checks whether the code is in compliance with CLAUDE.md though
Re:Handbook.md shows that long policy documents do not relia... by QuercusMax on Wednesday July 29, 2026 @11:53AM (#49099120)
I've seen pretty much every model including Claude bypass commit hooks constantly if there's even a tiny bit of friction, instead of pausing and asking me to help fix why it can't run the hooks (usually something that needs to be done outside the sandbox like `npm ci`).

That's why it's critical to have these checks run in a context that the model can't bypass, such as via github actions that block PR merges. Generally works pretty well for stuff at work (where we have all this stuff set up as part of continuous integration checks), but it means that for personal stuff I have to set up quite a bit more infrastructure to ensure that models don't just skip running tests (which they LOVE to do).

Re:Handbook.md shows that long policy documents do not relia... by pmarreck on Wednesday July 29, 2026 @06:37PM (#49103996)
Or just make it so that the commit hook can't be edited or bypassed by Claude?
Re:Handbook.md shows that long policy documents do not relia... by PunchyHamster on Wednesday July 29, 2026 @10:45AM (#49098231)
So you are essentially saying "you can't, you can only safeguard from effects of LLM eventually ignoring it"
Re:Handbook.md shows that long policy documents do not relia... by dbalatero on Wednesday July 29, 2026 @11:09AM (#49098548)
I think things like CI, linters, hooks, etc started as defense against humans eventually ignoring it (low blood sugar, under pressure, never cared in the first place, etc).

These seem like they would naturally extend to LLMs as well.

(I'm not pro-LLM on balance, but the solutions of today feel similar for humans and LLMs.)

Re:Handbook.md shows that long policy documents do not relia... by pmarreck on Wednesday July 29, 2026 @06:38PM (#49104001)
It is almost exactly like the controls from the auditing world, except as applied to AI's instead of people. In fact, having worked at Deloitte in the past is what led me to make the connection immediately
Re:Handbook.md shows that long policy documents do not relia... by pmarreck on Wednesday July 29, 2026 @05:16PM (#49103201)
Just like humans, yes.