nvmdbljstm

How many cats were used to build this?

Incredible and horrifying at the same time

~3x cheaper and ~3x quicker

Good point—though, tbh I’ve never benchmarked this myself. I just looked at the RTK gain, and it seemed to do the job." That said, because of my specific sandbox configuration, I’ve added gh and git to my "excluded tools" list, so I’m in no position to compare it against gh-axi.

Here’s an example of gh-axi, which wraps the GitHub CLI and prepares the most relevant information for agents—so the LLM doesn’t have to filter it out itself. So simple! I haven’t yet formed an opinion on whether this is overengineering or a genuine way to save time and money; I also haven’t looked into how it compares to the cost savings provided by RTK.

Thanks a lot for your perspective. I needed a comparison cause I only used Phoenix. Its architecture is extremely simple. Tracing is nice, I didn't have any complaints. All other features like evaluations, datasets, prompts - I'd use a separate tool for this.

Regardless of whether it’s useful or not, the level of creative intelligence is impressive

How do the principles of traditional Quality Engineering apply to systems built on top of LLMs? In the article, I look at what it takes to move from a deterministic approach to a probabilistic one in modern AI delivery.

How was your experience with reliability, turnaround time, and cost efficiency? Any issues or lessons learned? How it compares to Claude Batch API?

do you have any mods yet?

I show how to evaluate AI systems as a whole — model, harness, tools, memory, execution path, reliability, cost, and business outcome — under production-like conditions.