viktorianer

If you have no feedback for 6 minutes, it will hold up your time.

Rails is a good example of why. One correct place for migrations. One expected controller structure. Validations on the model, callbacks with a defined lifecycle. When the framework defines where things belong, you can write automated rules that catch bad AI output: "no business logic in controllers," "no conditional assertions in tests." Each rule works because Rails narrows what "correct" means.

Frontend has no equivalent constraint. A component can be structured a hundred different ways, and "does it look right" requires vision the model does not have.

The freed-up time question is answerable when the work has clear metrics. A model test suite dropping from 6 minutes to 66 seconds saves developer time on every single run. Ten developers running tests five times a day, the math is straightforward.

The problem is that most engineering work lacks that kind of before/after measurement. Not because it is unmeasurable, but because nobody set up the baseline. Profile before you optimize and the return on investment calculates itself.

The hallucinated GUIDs are a class of failure that prompt instructions will never reliably prevent. The fix that worked: regex patterns running on every file the agent produces, before anything executes.

I went the compensation route: a set of regex patterns that check generated code for known anti-patterns before it runs. The agent can ignore a prompt telling it "don't use mocks here." It cannot ignore a failed exit code from a pattern check. Started with 40 patterns and grew to 138 over a few months as new failure modes appeared.

The false comfort usually comes from line coverage. I had a model at 97.2% coverage, 92 tests. Ran equivalence partitioning on it (partition inputs into classes, test one from each) and found 6 real gaps: a search scope testing 1 of 5 fields, a scope checking return type but not filtering logic, a missing state machine branch. SimpleCov said the file was covered. The logical input space was not. The technique is old (ISTQB calls it specification-based testing) but the manual overhead made it impractical until recently. Agents made it possible to apply across 60+ models, which is the one thing they have changed for testing so far.

The fix: the agent that writes code never gets to judge its own output. A separate script scans for known bad patterns, things like reduced test count or forbidden syntax. If the scan fails, the agent loops. It cannot argue with an exit code.

Exactly right. Files on disk as the shared state, not the conversation window. Each step reads current state, does its job, writes output. Next step starts fresh from those files. No accumulated context means no drift, and the LLM can hallucinate in its reasoning all it wants as long as the output passes a check before anything advances.

Agree completely. The middle ground between "please don't" and full sandboxing: run a validation script between agent steps. The agent writes code, a regex check catches banned patterns, the agent has to fix them before it can proceed. Sandboxing controls what the agent can do. Output validation controls what it gets to keep. Both are more reliable than prompt instructions.

Same experience. I told my agents "never use FactoryBot, always use fixtures." They followed it most of the time. Then under pressure (complex model, lots of associations) they'd fall back to FactoryBot anyway. The agent "knew" the rule. It just didn't always follow it.

What actually worked: a Ruby script that scans the output for known bad patterns. If it finds FactoryBot in the generated code, the agent can't proceed. No reasoning, no judgment call, just a regex match and a hard stop. The agent fixes the violation, then tries again. Went from 40 patterns to 138 over 98 model runs as new failure modes showed up.

I ran a Claude Code pipeline across 98 Rails models, 9,000 tests total. The thing that made it repeatable: each agent can only do one job. The analyzer writes a YAML plan but no Ruby. The writer gets one slice of that plan, not the whole thing. A Ruby script (not the AI) checks the output for 138 known mistakes before anything moves forward. If the check fails, the agent has to fix it.

That Ruby script is where the data comes from. I can see exactly which checks caught which problems across all 98 runs. That is not anecdotal, it is a log file.

I ran a multi-agent system that generated tests for 60+ Rails models. One spec: 259 examples, 70% of time in factory creation, 897 factory calls. Mathematically rigorous tests that spent most of their time waiting for the database.

AI does not validate its own output. My system runs 138 regex-based checks before an agent can commit. Zero tokens, zero LLM reasoning. The same tooling turned out useful for humans too. The real gamble is using AI without deterministic quality gates between generation and commit.