How do you keep AI-generated applications consistent as they evolve over time?

Re:How do you keep AI-generated applications consistent as t... by RobertSerber on Saturday February 07, 2026 @08:14AM (#46923596)

This makes a lot of sense — especially the “LLMs as nondeterministic services” framing. I agree that versioned prompts, schema validators, regression evals, and contract tests are essential if you're shipping LLM-powered systems.

What I’m wrestling with is a slightly different failure mode though. Even if: prompts are versioned outputs conform to JSON schema contract tests pass …you can still end up with semantic drift inside the application model itself.

Example: Entity field renamed Metric still references old semantic meaning Relationship cardinality changes Permission scope shifts All technically valid JSON. All passing structural validation.

But the meaning of the system changes in ways that silently break invariants.

So the question becomes: Should lifecycle control live only at the “LLM output quality” layer (prompts, evals, CI), or does the runtime itself need semantic version awareness and invariant enforcement?

In other words: Even if the AI is perfectly guarded, do we still need an application-level planner that understands entities, metrics, relationships, and permissions as first-class concepts — and refuses inconsistent evolution? I am definitively into this.

Curious how you think about that boundary.

[ Parent | on HN ]

Re:How do you keep AI-generated applications consistent as t... by jredwards7 on Sunday February 08, 2026 @10:42AM (#46935252)

The structural validation layer is worth separating from the semantic layer, even if the semantic problem is harder. If every AI-proposed change is validated against a JSON Schema before execution, and that validation produces a deterministic, signed result, you get an auditable record of what passed structural checks and when. That doesn't solve semantic drift (a field can be renamed and still pass validation) but it gives you a foundation to build semantic checks on top of. The key property is determinism: identical schema and payload should produce identical hashes and outcomes, regardless of which service runs the check. Without that, you end up debugging whether a failure is in your schema logic or in validator implementation differences across languages. Contract tests and regression evals work better when the structural layer underneath is provably consistent. One reference for this pattern: https://docs.rapidtools.dev/openapi.yaml

[ Parent | on HN ]

Re:How do you keep AI-generated applications consistent as t... by RobertSerber on Sunday February 08, 2026 @12:33PM (#46936523)

Totally agree. We treat structural validation as a separate, deterministic “compiler phase” before anything executes.

Draft (AI output) → normalize/canonicalize → link/type-check (cross-refs, relationship/FK semantics, datasets/metrics) → append-only migration ops → canonical DSL + stable hash.

This is also why we’re building it as a compiler pipeline rather than “prompting harder”: the output is an auditable artifact (canonical DSL + hash + migration checksum) that makes contract tests/regression diffs trustworthy.

Next step is what you call out: a signed attestation per compile (schemaHash + pinned validator/version/options + canonicalHash), so identical input yields identical outcomes across services/languages.

You’re right it doesn’t solve semantic drift alone (renames can still validate), but once the deterministic floor exists we can layer semantic checks + rename-safe migrations on top without debugging the toolchain itself.

[ Parent | on HN ]