News for hackers. Stuff that matters.

hipvlady

Re: on Tuesday August 18, 2026 @11:09AM (#49346786)
I am the author and would like to point out that this mechanism is not new, since Kleppmann uses a similar approach with fencing tokens in his article "How to do distributed locking". The alternative I mention is also commonly employed. What interests me is examining the specific question each method addresses.

The situation in which things go wrong is this: a worker has been given a grant, becomes stuck on a slow call, loses the lease, and then carries out the commit. If no other writer has accessed that record during the period of stagnation, the version remains unchanged, and a simple CAS operation still functions, with the late write proceeding as before. This does not constitute a contention bug, so making the lock stricter helps little; that is usually what confuses. Although there is only one writer, its actions are divided between the stall and the reclaim.

A per-record generation number is used, incremented each time ownership changes, and this generation number is verified with the version when a commit is made. Two points I would have overlooked without guidance are these: if the generation number is only incremented during reclamation, then there is a gap. Since a clean release followed by a new claim keeps the same generation number, a late duplicate commit could still match both numbers. Also, the check has to happen at the same time as the write; otherwise you'd merely shift the race condition rather than eliminate it.

A frequent objection I respond to directly is to replace the version stored on reclaim with an empty write. In this case, a basic CAS will detect it without a second counter, and this method works. The drawback is that one counter then has to answer two different questions. As a result, every reader's cache is invalidated by a write that didn't actually change anything, and a writer whose request is rejected sees only a general version conflict rather than a clear indication that their grant has expired. This matters because the recovery procedures are different.

As for scope, since this group will examine it, there is one coordinator and one host, along with the writers who commit through it. I prefer to make my own concessions regarding vocabulary rather than have them made for me. When the grant authority and the commit path are in the same place, an equality check on a generation is simply the degenerate case of a fencing token. Fencing tokens exist exactly because the store is generally unable to ask the lock service anything. The case involving multiple hosts, where the check becomes the highest token seen at a store that is not the grant authority, is precisely the part that has not yet been shipped.

The protocol, not the actual code, is modeled using TLA+. I employ bounded model checking, and for each specification that should fail, I create a documented mutant. I never state that anything has been 'proved'.

I would like to know what failure modes in worker pools this method might miss.

Author here. Why do we need a taxonomy when we could just say "we stop lost updates"?

Two of the four aren't mine to fix. Key mismatch (write to X, read from Y) and compaction drop (a summarisation step eats the fact) are application-layer bugs. The most a layer underneath can do is make them visible in a trace. I separated them because mixing them up with the preventable ones is how people end up "fixing" a race that was actually a routing bug.

The two that are preventable are the classic ones, just with new names. Concurrent lost-update is a read-modify-write race, handled with a compare-and-set on a version that travels with the write. If you read something and then write something, it's called 'stale-read-then-write'. This is dealt with using something called 'invalidation'. This means that the stale writer is denied and forced to re-read before it can land anything. None of the mechanisms are new. MESI is from CPUs and CAS is from databases. The only thing they need to do is connect them to the write paths that the agent frameworks use, as well as the TLA+ specifications that run in CI with a mutant per specification that has to fail.

The plan is simple: there will be one coordinator, one host, and writers who will work with them. Cross-host isn't available yet; it's on a demand basis. I'd really like to hear about any other failure modes.

Great review. If you use this program every day, there is one thing you should know. CLAUDE.md is doing a lot of the most important work here, and it's the most fragile piece of the stack. A rule like "always re-read X before editing" is written as a piece of prose, and prose is subject to context. It's either one compaction or one subagent that appears from quietly disappearing, and there's no warning when this happens. The subagent case is the clear one: a subagent gets a new window, doesn't have the parent's CLAUDE.md discipline, and acts on the assumption that the rule was meant to stop something. The only thing that can reliably enforce the rules is settings.json permissions.deny, which the runtime checks before the model picks a tool. This means that you can't use cat or grep to bypass it. Everything left in prose is a strong default, not a guarantee. It's good to know which of your rules are which.
Re:My Seven Claude Code Subagents Did the Easy Part on Thursday June 04, 2026 @10:03AM (#48398818)
The title is very fashionable at the moment. Subagents are good at the part that can be broken down into separate tasks. The problem is that each subagent has its own understanding of the plan. When the plan changes, or if one subagent changes its assumptions, the windows don't stay in sync. You get work that's similar but doesn't fit together, and this shows up when you're integrating. The easy part is parallelism. The difficult part is making sure the different agents work well together, and that's not something you can easily achieve by increasing the number of subagents. I'm curious to know if your seven were reading from one source that stayed fixed, or if any of them were supposed to react to what the others produced.
A shared handoff file is better than implicit coordination. The problem to watch out for is the file becoming out of date: agent A reads Agents-Collab.md, agent B updates it mid-task, and agent A continues to act on what it has read. The protocol is only as good as how fresh each agent is on the file that contains it. "Live" is the hard part when two agents run at once instead of handing off one after the other. If each agent had a version or timestamp that they checked before they acted, it would stop a handoff written after A's last read from being ignored.
Saying "agents lying" isn't usually deception. It's the model telling a story about a gap that the runtime couldn't fill. This gap could be a rule or a state that the runtime was supposed to respect, but wasn't actually enforced. This means that the contract is completed as text. It is best to use a harness that enforces the SOP out of the model's reach. This is because if you leave anything as prose in context, it will lead to one compaction or one subagent spawn from evaporating. The most difficult case to test would be an SOP that depends on a state that changes under the agent. If step 3 says "use the latest config" and another process updated it after the agent read it at step 1, will the harness re-check, or enforce against the snapshot the agent loaded?
Re: on Tuesday June 02, 2026 @12:11PM (#48372196)
It's really useful to have a clear list of who's on the team and what they do, because there isn't one at the moment. A static team definition doesn't show the runtime hazard. This is when two agents read the same artifact. One updates it, while the other keeps acting on the version it loaded. The roles can be clearly defined, but the output is still unclear because agent B wasn't told that agent A had moved the shared state. I would like to know if you think that coordination layer is part of the problem, or if it is only a problem with the runtime.
It wasn't a crash that cost us the most time; it was a stale read. Two agents shared a plan file. One updated the file during the run and the other continued to work off the version that had been loaded at the start. Both produced plausible output. Nothing errored. We only discovered this issue during the review process, after wasting hours of generation time. I now treat any artefact that two agents can both access as a coordination problem, not a storage one. The cheapest and most effective solution was to stamp a version on the artefact and re-check it before acting, instead of trusting the copy read at the start of the run.