nyellin

Author of the original here. Thank you for the mention!

No, because every article will devolve into a discussion on if AI generated or not, and to what degree. Let content speak for itself.

OP here and the doctor was wonderful. If we fixed the interface they could help more patients in the same time.

It's a trade-off. In these scenarios we update our staging data for next time.

How does copy pasting an error into an LLM involve thinking?

I'm the OP and wanted to clarify. We do not give Claude Code access to production API keys. The intent was staging databases and I've updated the post to make it more clear.

We assume that every API key we put into Claude Code can be leaked and therefore make sure there's nothing sensitive behind them.

I'm the OP and this is the exact point of the post. Whoever handed you that project did not have Claude code running it end to end in a production-like environment.

Whether you like the vibe coded app or not, you would have had fewer errors if Claude had been testing it end to end while vibe coding.

As written elsewhere, we dont give access to prod! The DBs are staging and our assumption is that every key we give Claude will be leaked. I'll update the post to clarify

Fair enough, I will update

I'm the OP and to clarify we dont give access to prod DBs. The point is you need to give the LLM the ability to test end to end, and that can be done with staging data.

OP here: we don't give Claude Code access to prod. Everything is isolated cloud accounts for this purpose.

E.g. we give Claude credentials for db - but it's never prod data.

No, it was designed on paper by someone with no understanding of prompt caching and no consideration of latency or token costs

Why does Atlassian need to train AI models?

Is it possible to run a Kubernetes cluster inside one? (E.g. via KIND.)

If so, we'd very much like to test this. We make extensive use of Claude Code web but it can't effectively test our product inside the sandbox without running a K8s cluster

We publish the benchmarks for HolmesGPT (CNCF sandbox project) at https://holmesgpt.dev/development/evaluations/

HolmesGPT maintainer here: our benchmarks [1] tell a very different story, as does anecdotal evidence from our customers- including Fortune 500 using SRE agents in incredibly complex production environments.

We're actually struggling a bit with benchmark saturation right now. Opus does much better in the real world than Sonnet but it's hard to create sophisticated enough benchmarks to show that in the lab. When we run benchmarks with a small number of iterations Sonnet even wins sometimes.

[1] https://holmesgpt.dev/development/evaluations/history/

Haiku is called often, but not always the way you think. E.g. every time you write something CC invokes Haiku multiple times to generate the 'delightful 1-2 word phrase used to indicate progress to the user' (Doing Stuff, Wizarding, etc)

Not necessarily true. Subagents allow for parallelization but they can decrease accuracy dramatically if you're not careful because there are often dependencies between tasks and swapping context windows with a summary is extremely lossy.

For the longest time, Claude Code itself didnt really use subagents much by default, other than supporting them as a feature eager users could configure. (Source is reverse engineering we did on Claude code using the fantastic CC tracing tool Simon Willison wrote about once. This is also no longer true on latest versions that have e.g. an Explore subagent that is actively used.)

Forgot to address the easiest part:

> - how can I reliably call tools with the right schema?

This is typically done by enabling strict mode for tool calling which is a hermetic solution. Makes llm unable to generate tokens that would violate the schema. (I.e. LLM samples tokens only from the subset of tokens that lead to valid schema generation.)

Re (1) use a TODOs system like Claude code.

Re (2) also fairly easy! It's just a summarization prompt. E.g. this is the one we use in our agent: https://github.com/HolmesGPT/holmesgpt/blob/62c3898e4efae69b...

Or just use the Claude Code SDK that does this all for you! (You can also use various provider-specific features for 2 like automatic compaction on OpenAI responses endpoint.)

There's a bit more to it!

For example, the agent in the post will demonstrate 'early stopping' where it finishes before the task is really done. You'd think you can solve this with reasoning models, but it doesn't actually work on SOTA models.

To fix 'early stopping' you need extra features in the agent harness. Claude Code does this with TODOs that are injected back into every prompt to remind the LLM what tasks remain open. (If you're curious somewhere in the public repo for HolmesGPT we have benchamrks with all the experiments we ran to solve this - from hypothesis tracking to other exotic approaches - but TODOs always performed best.)

Still, good article. Agents really are just tools in a loop. It's not rocket science.

I know there are already a number of comments here about proprietary solutions.

If you're looking for something open source: https://github.com/robusta-dev/holmesgpt/

Here you go: https://www.youtube.com/live/Jml1hk6I5Wo?si=YbjJKRkO4yf0bOlx

And thanks for submitting!

We've open sourced something with similar goals that you can use today: https://github.com/robusta-dev/holmesgpt/

We're taking a slightly different angle than what Facebook published, in that we're primarily using tool calling and observability data to run investigations.

What we've released really shines at surfacing up relevant observability data automatically, and we're soon planning to add the change-tracking elements mentioned in the Facebook post.

If anyone is curious, I did a webinar with PagerDuty on this recently.