behat

Daughter loves getting these in the mail, and this paragraph is absolutely right!

>> That year, Parton explained that as well as improving child literacy, “it’s the fact they get recognised. They get this little book with their little name on it in the mail, and they feel special. They start taking pride in themselves, and they know that somebody out there is thinking of [them].”

Appreciate the detail in this and the previous post on creating internal benchmarks!

Have you all attempted finetuning smaller OSS models on your repos for coding?

well, at least they are not leaving to be MTS at Anthropic.

Thanks for writing this, and look forward to the one on procedural memory.

Seems like teams are encoding procedural knowledge in skills repositories, and I wonder if there’s additional utility from an auto created procedural memory layer

>> Superlog scans your codebase and infrastructure to add new alerts, metrics and dashboards, preventing tricky failure modes and observability decay.

This is interesting, and my prior belief here has been that this automates a one time set up, and perhaps a quarterly clean-up or reactive monitoring changes that people do today. Curious what your experience has been - do teams accept these ongoing maintenance PRs at a good rate?

For full disclosure / context: we work in a related space - investigation agents for production issues.

This is a very interesting read on failure modes of AI agents in prod.

Curious about this section on the system prompt change: >> After multiple weeks of internal testing and no regressions in the set of evaluations we ran, we felt confident about the change and shipped it alongside Opus 4.7 on April 16. As part of this investigation, we ran more ablations (removing lines from the system prompt to understand the impact of each line) using a broader set of evaluations. One of these evaluations showed a 3% drop for both Opus 4.6 and 4.7. We immediately reverted the prompt as part of the April 20 release.

Curious what helped catch in the later eval vs. initial ones. Was it that the initial testing was online A/B comparison of aggregate metrics, or that the dataset was not broad enough?

>> how the runbooks can self heal if results from some steps in the middle are not expected.

Yeah this is a very interesting angle. Our primary mechanism here is via agent created auto-memories today. The agent keeps track of the most useful steps, and more importantly, dead end steps as it executes runbooks. We think this offers a great bridge to suggest runbook updates and keep them current.

>> Curious how much savings do you observe from using runbook versus purely let Claude do the planning at first.

Really depends on runbook quality, so I don't have a straightforward answer. Of course, it's faster and cheaper if you have well defined steps in your runbooks. As an example, `check logs for service frontend, faceted by host_name`, vs. `check logs`. Agent does more exploration in the latter case.

We wrote about the LLM costs of investigating production alerts more generally here, in case helpful: https://relvy.ai/blog/llm-cost-of-ai-sre-investigating-produ...

Do you mean how we connect to internal data? Today, you can connect any API endpoint to Relvy, so if you have internal business data / dashboards that you look at while debugging, Relvy can do the same if there's an API for it.

Most of our deployments are self-hosted, in which case the data stays locally (your chosen LLM provider exempted), if that's what you are asking.

Nice to see you here, Will! I’d generally recommend using open telemetry for instrumentation so that you keep the option of switching between telemetry vendors.

Re: runbooks, yeah even larger teams don’t have good ones to begin with. Relvy helps debug without runbooks as well - it might take longer to explore, but once you are happy with a particular investigation path the AI took, you can save it as a runbook for more deterministic future executions.

> They want extension to their agent. If a project tells me I have to use their interface or agentic setup, it's 95% not going to happen

Yes, there’s definitely friction there. It may be that the right form factor is that you trigger Relvy’s debugging agent via Claude code / Cursor .

Our early users are heavy on needing to look at the raw data to be able to review the AI RCA, so a standalone set up makes sense. Also, the dominant usage pattern is background agentic execution triggered by alerts, and not manual.

Yes! That boundary between what can be automated and what still needs human judgement has shifted so much this last year. Things like 'go check this dashboard' can now be automated.

ROI on runbooks (or good documentation in general) is much higher now if you have AI agents running them autonomously in the background. Makes it worth it to write/maintain runbooks.

heh, I was just about to post the following on your previous comment re: reproducible benchmark results. Thanks for posting the blog.

With the docker images that we offer, in theory, people can re-run the benchmark themselves with our agent. But we should document and make that easier.

At the end of it, you really would have to evaluate on your own production alerts. Hopefully the easy install + set up helps.

For the debugging workflow you described, we would be a standalone replacement for cursor or other agents. We don't yet write code so can't replace your cursor agents entirely.

Re: diffentiation - yes, faster, more accurate and more consistent. Partially because of better tools and UX, and partially because we anchor on runbooks. On-call engineers can quickly map out that the AI ran so-and-so steps, and here's what it found for each, and here's the time series graph that supports this.

Interesting that you have had great success with Datadog MCP. Do you mainly look at logs?

Thank you :)

Thanks. Yeah, Cursor / Claude code + MCP is powerful. We differentiate on two fronts, mainly:

1) Greater accuracy with our specialized tools: Most MCP tools allow agents to query data, or run *ql queries - this overwhelms context windows given the scale of telemetry data. Raw data is also not great for reasoning - we’ve designed our tools to ensure that models get data in the right format, enriched with statistical summaries, baselines, and correlation data, so LLMs can focus on reasoning.

2) Product UX: You’ll also find that text based outputs from general purpose agents are not sufficient for this task - our notebook UX offers a great way to visualize the underlying data so you can review and build trust with the AI.

An on-call runbook execution engine - being able to take plain text runbook steps like - look at logs for $service, check for dependency failures - look at so-and-so dashboard.

and execute them when an alert fires.

We are at https://www.relvy.ai

Heh. Built a macOS app that does something like this a while ago - https://github.com/bharathpbhat/EssentialApp

Back then, I used on device OCR and then sent the text to gpt. I’ve been wanting to re-do this with local LLMs

Nice! Built something similar earlier to get fixes from chatgpt for error messages on screen. No voice input because I don't like speaking. My approach then was Apple Computer Vision Kit for OCR + chatgpt. This reminds me to test out OpenAI's Vision API as a replacement.

Thanks for sharing!

Thank you for sharing! The HF docs seem easy to follow. My application is text generation itself, so may have different results.