Considering it as a classification problem, you could use a representative set of example queries, feed them to this model, then ask a "smart" language model to assess each query + result for whether the result is actually correct. Then you have a dataset you can compute FPR and FNR for any given threshold, and score based on the context, ex. turning the lights on or off is a lot less important if it is right compared to whether your door is locked. You could even pick it based on the tool call itself: Low threshold for locking the door, high threshold for unlocking the door.