whysosuresh
Do you which one it was?
What helped me is to look at this problem as two parts: 1. Whether the agent is making the right decision. 2. Does the real system respond the way it should be?
To test the first part, we don't need the real system. We can build a fake system with the tool interfaces, give it to the agent, and see if the agent is making the right decisions or not.
Also, we need to make sure that the fake system is not that perfect. We should inject issues like timeouts, wrong data format, not returning, or returning a half answer, to make sure that the agent works in all these different situations, not only the positive flow.
And even after this testing to make sure that it works correctly in production, I recommend you enforce a human in the loop before making any state changes. Keep a watch on these changes for some time and see if the agent is consistently doing the right thing before relaxing the controls.
Also, I would like to know whether this system that you can't reach is a third-party product or a customers environment. That answer changes a lot.
Also, is the instant switch being done through private Windows server calls, or did you find something public?
For example, I built a small Mac app called CapsJump to set up shortcuts for Chrome profiles. In one version, it broke for users who use multiple displays, and it was bringing up all the browser windows instead of the one they wanted. In another version, it broke the simple setup for someone with just one browser profile.
For me, it was mostly identifying those unique setups and trying to fix them.
I've started using Stryker. We have a CI job that mutates the code before running the tests. In the first run, it has returned hundreds of code changes confirming that the tests are biased. I haven't yet fixed those issues, but for now i am banking on this approach.