abdullin

They are flavor of BDD specs. More data than the code.

Either way Codex writes them under my supervision.

20+ years of software experience.

Coding part of the job (as in „typing with my bare hands“) is gone. Now I get to spend more time making product decisions, capturing constraints and executable specs. Resulting code is boring, consistent and gets the job done.

I’m struggling myself a bit with the components and routers in HTMX/go. Can you recommend any source code or reading on that topic?

#1 - the tricky part there is in scenarios from a few AI Native teams. There often are a multiple agents rolling out linked changesets to a bunch of documents on behalf of controlling humans. Eg updating compliance policy, and references and change log and current procedures at the same time.

So changesets have to be atomic across multiple documents and semantic (so that agents can resolve the changes). Weak per-document versioning isn’t enough here.

#4. Nice! Same story, but also virtualizing ripgrep, find and tree (plus MD-aware outline mode). With that setup even agents with weaker local models (eg runnable on DGX Spark) can solve complex tasks in the Agentic Commerce domain.

Nice approach.

Personally I’ve been trying very hard to migrate away from git+Obsidian project setup according to the OpenAI Harness Engineering. It works wonderfully in Codex Desktop.

The only gotcha - I want to share knowledge bases with the team in a way that is:

(1) versioned (a la git, not Notion) (2) usable from any chat (a la MCP) (3) basic access controls for team setup. (4) works through the interface that optimizes accuracy and token use across agentic architectures and LLMs.

Funnily enough, 4 is the easiest one (I have a platform for agent training and verification where I publish fun challenges for agents in simulated worlds around agentic commerce and personal OSes. With 98M agentic interactions recorded, that is already enough information for tuning)

Still figuring 1 and 3, though.

Working on benchmark arena for AI agents with my wife.

We grab interesting business problems, turn them into fun challenges for hundreds of AI engineers to find the best architecture for. Insights are shared back with the community.

It is a fun learning process with unexpected scaling challenges.

I reproduced this on my account.

    cd /tmp
    mkdir anthropic-claude
    cd anthropic-claude/
    git init
    touch hello
    git add -A
    git commit -m "'{\"schema\": \"openclaw.inbound_meta.v1\"}'"
    claude -p "hi"

Immediate disconnect and session usage went to 100%

I built a platform to learn how to build personal AI agents and test them with fast feedback. It is free for individuals and small teams.

Platform deterministically generates tasks, creates environments for them, observes AI agents and then scores them (not LLM as a judge).

We just ran a worldwide hackathon (800 engineers across 80 cities). Ended up creating more than 1 million runtimes (each task runs in its own environment) and crashing the platform halfway.

104 tasks from the challenge on building a personal and trustworthy AI agent are open now for everyone.

https://bitgn.com/

To get started faster you can use a simple SGR Next Step agent: https://github.com/bitgn/sample-agents

I liked NixOS pre-LLM era, since it allowed me to manage a couple of servers in a reproducible way. Ability to reboot back to a stable configuration felt like magic.

Nowadays I love it, since I can let Codex manage the servers for me.

“Here is the flake, here is nix module for the server, here is the project source code. Now change all of that so that wildcard certificates work and requests land through systemd socket on a proper go mux endpoint. Don’t come back until you verify it as working”

5 minutes later it came back.

Yep, exactly the same concept. Except not live-streaming, but giving out a lot of multi-step tasks that require reasoning and adaptation.

Here is a screenshot of a test task: https://www.linkedin.com/posts/abdullin_ddd-ai-sgr-here-is-h...

Although… since I record all interactions, could replay all them as if they were streamed.

I’m working on a platform to run a friendly competition in “who builds the best reasoning AI Agent”.

Each participating team (got 300 signups so far) will get a set of text tasks and a set of simulated APIs to solve them.

For instance the task (a typical chatbot task) could say something like: “Schedule 30m knowledge exchange next week between the most experienced Python expert in the company and 3-5 people that are most interested in learning it “

AI agent will have to solve through this by using a set of simulated APIs and playing a bit of calendar Tetris (in this case - Calendar API, Email API, SkillWill API).

Since API instances are simulated and isolated (per team per task), it becomes fairly easy to automatically check correctness of each solution and rank different agents in a global leaderboard.

Code of agents stays external, but participants fill and submit brief questionnaires about their architectures.

By benchmarking different agentic implementations on the same tasks - we get to see patterns in performance, accuracy and costs of various architectures.

Codebase of the platform is written mostly in golang (to support thousands of concurrent simulations). I’m using coding agents (Claude Code and Codex) for exploration and easy coding tasks, but the core has still to be handcrafted.

> Inference is (mostly) stateless

Quite the opposite. Context caching requires state (K/V cache) close to the VRAM. Streaming requires state. Constrained decoding (known as Structured Outputs) also requires state.

Is it similar to what OpenAI Codex does with isolated environments per agent run?

In systems like that you can record human interactions with the old version, replay against the new one and compare outcomes.

Is there a delta? Debug and add a unit test to capture the bug. Then fix and move to the next delta.

I grew to like migration projects like that.

Currently working on migration of 30yo ERP without tests in Progress to Kotlin+PostgreSQL.

AI agents don’t care which code to read or convert into tests. They just need an automated feedback loop and some human oversight.

I think there are two different layers that get frequently mixed.

(1) LLMs as models - just the weights and an inference engine. These are just tools like hammers. There is a wide variety of models, starting from transparent and useless IBM Granite models, to open-weights Llama/Qwen to proprietary.

(2) AI products that are built on top of LLMs (agents, RAG, search, reasoning etc). This is how people decide to use LLMs.

How these products display results - with or without citations, with or without attribution - is determined by the product design.

It takes more effort to design a system that properly attributes all bits of information to the sources, but it is doable. As long as product teams are willing to invest that effort.

Yes. I believe, the experience will get better. Plus more AI vendors will catch up with OpenAI and offer similar experiences in their products.

It will just take a few months.

Here is another way to look at the problem.

There is a team of 5 people that are passionate about their indigenous language and want to preserve it from disappearing. They are using AI+Coding tools to:

(1) Process and prepare a ton of various datasets for training custom text-to-speech, speech-to-text models and wake word models (because foundational models don't know this language), along with the pipelines and tooling for the contributors.

(2) design and develop an embedded device (running ESP32-S3) to act as a smart speaker running on the edge

(3) design and develop backend in golang to orchestrate hundreds of these speakers

(4) a whole bunch of Python agents (essentially glorified RAGs over folklore, stories)

(5) a set of websites for teachers to create course content and exercises, making them available to these edge devices

All that, just so that kids in a few hundred kindergartens and schools would be able to practice their own native language, listen to fairy tales, songs or ask questions.

This project was acknowledged by the UN (AI for Good programme). They are now extending their help to more disappearing languages.

None of that was possible before. This sounds like a good progress to me.

Edit: added newlines.

It is actually funny that current AI+Coding tools benefit a lot from domain context and other information along the lines of Domain-Driven Design (which was inspired by the pattern language of C. Alexander).

A few teams have started incorporating `CONTEXT.MD` into module descriptions to leverage this.

Agreed. AI is just a tool. Letting in run the show is essentially what the vibe-coding is. It is a fun activity for prototyping, but tends to accumulate problems and tech debt at an astonishing pace.

Code, manually crafted by professionals, will almost always beat AI-driven code in quality. Yet, one has still to find such professionals and wait for them to get the job done.

I think, the right balance is somewhere in between - let tools handle the mundane parts (e.g. mechanically rewriting that legacy Progress ABL/4GL code to Kotlin), while human engineers will have fun with high-level tasks and shaping the direction of the project.

Exactly!

This is why there has to be "write me a detailed implementation plan" step in between. Which files is it going to change, how, what are the gotchas, which tests will be affected or added etc.

It is easier to review one document and point out missing bits, than chase the loose ends.

Once the plan is done and good, it is usually a smooth path to the PR.

I guess, it depends on the case and the approach.

It works really nice with the following approach (distilled from experiences reported by multiple companies)

(1) Augment codebase with explanatory texts that describe individual modules, interfaces and interactions (something that is needed for the humans anyway)

(2) Provide Agent.MD that describes the approach/style/process that the AI agent must take. It should also describe how to run all tests.

(3) Break down the task into smaller features. For each feature - ask first to write a detailed implementation plan (because it is easier to review the plan than 1000 lines of changes. spread across a dozen files)

(4) Review the plan and ask to improve it, if needed. When ready - ask to draft an actual pull request

(5) The system will automatically use all available tests/linting/rules before writing the final PR. Verify and provide feedback, if some polish is needed.

(6) Launch multiple instances of "write me an implementation plan" and "Implement this plan" task, to pick the one that looks the best.

This is very similar to git-driven development of large codebases by distributed teams.

Edit: added newlines

Running tests is already an engineering problem.

In one of the systems (supply chain SaaS) we invested so much effort in having good tests in a simulated environment, that we could run full-stack tests at kHz. Roughly ~5k tests per second or so on a laptop.

Claude's approach is currently a bit dated.

Cursor.sh agents or especially OpenAI Codex illustrate that a tool doesn't need to keep on stuffing context window with irrelevant information in order to make progress on a task.

And if really needed, engineers report that Gemini Pro 2.5 keeps on working fine within 200k-500k token context. Above that - it is better to reset the context.

A simple rule applies: "No matter what tool created the code, you are still responsible for what you merge into main".

As such, task of verification, still falls on hands of engineers.

Given that and proper processes, modern tooling works nicely with codebases ranging from 10k LOC (mixed embedded device code with golang backends and python DS/ML) to 700k LOC (legacy enterprise applications from the mainframe era)

Humans tend to lack inhumane patience.

This is the workflow that ChatGPT Codex demonstrates nicely. Launch any number of «robotic» tasks in parallel, then go on your own. Come back later to review the results and pick good ones.

Tight feedback loops are the key in working productively with software. I see that in codebases up to 700k lines of code (legacy 30yo 4GL ERP systems).

The best part is that AI-driven systems are fine with running even more tight loops than what a sane human would tolerate.

Eg. running full linting, testing and E2E/simulation suite after any minor change. Or generating 4 versions of PR for the same task so that the human could just pick the best one.

Even more than that. With Structured Outputs we essentially control layout of the response, so we can force LLM to go through different parts of the completion in a predefined order.

One way teams exploit that - force LLM to go through a predefined task-specific checklist before answering. This custom hard-coded chain of thought boosts the accuracy and makes reasoning more auditable.