Re:Ask HN: Angents and Clsoed Platform
by itstomo on Wednesday November 18, 2026 @07:57PM
(#50554642)
Hi, thank you for the reply.
The system that we couldn't reach is a third-party product. They only provide the software and documentation to clients they have a contract with, so we couldn't get access to the real product before the demo.
We did use a mock environment like the one you described here. But our agent still failed in the demo call. Maybe our mock env was just not good enough.
Re:Ask HN: Angents and Clsoed Platform
by whysosuresh on Thursday November 19, 2026 @12:10AM
(#50556295)
Yes, it is important for us to understand where the problem is. Is it how the mock got set up, or the agent decision.
Do you which one it was?
What helped me is to look at this problem as two parts: 1. Whether the agent is making the right decision. 2. Does the real system respond the way it should be?
To test the first part, we don't need the real system. We can build a fake system with the tool interfaces, give it to the agent, and see if the agent is making the right decisions or not.
Also, we need to make sure that the fake system is not that perfect. We should inject issues like timeouts, wrong data format, not returning, or returning a half answer, to make sure that the agent works in all these different situations, not only the positive flow.
And even after this testing to make sure that it works correctly in production, I recommend you enforce a human in the loop before making any state changes. Keep a watch on these changes for some time and see if the agent is consistently doing the right thing before relaxing the controls.
Also, I would like to know whether this system that you can't reach is a third-party product or a customers environment. That answer changes a lot.