Jev-Driven SRE Diagnosis: What Worked and What Failed

Re:Jev-Driven SRE Diagnosis: What Worked and What Failed by endangeredhuman on Wednesday October 07, 2026 @02:19AM (#49989014)

This is an interesting experiment. Curious why you used LLM-as-a-Judge (gpt-6-astra). Did we have a ground truth of actual RCA done by a human to compare against?

[ Parent | on HN ]

Re:Jev-Driven SRE Diagnosis: What Worked and What Failed by yimingsu on Wednesday October 07, 2026 @11:33AM (#49994294)

Yes! All faults in SREGym-lite (and the parent benchmark SREGym) have their ground truths reviewed by us to ensure that they are accurate to the incident. In addition, we also conducted an analysis to ensure the LLM judge agrees with the human annotators too, which can be found on page 6 here: https://arxiv.org/abs/2605.07161.

[ Parent | on HN ]