First clarify what the number means
With Noul, you state a proposition and the model estimates its probability in the provided context. For example: “Under the rules, this request requires urgent human review.” The model assesses that specific statement, not the overall quality of an agent's work.
Do not automatically interpret a confidence value as a guaranteed future accuracy rate. Whether numerical estimates match observed outcomes must be checked on labeled examples of your task. New wording or a different language can change the picture.
Choose a rule based on the cost of errors
In support, unnecessary review takes an agent's time, while missing a widespread outage may delay the team's response. These consequences differ. Collect cases whose correct next step is known, then compare the errors and manual workload produced by each possible threshold.
Use different examples to choose a rule and evaluate it finally. A threshold chosen on a sample you have already examined does not prove it will work the same way on new traffic. If too many facts are unknown, asking for clarification may be the right next step regardless of the number returned.
- Identify the errors it is especially important not to miss.
- Estimate the human workload for each rule.
- Test the chosen rule on held-out examples.
Give the reviewer the decision context
Send the original context, question, result and rule version to the manual queue. People need the facts to reach their own conclusions. A confidence number without the question and criteria explains little.
Record confirmations or corrections with a brief reason. These examples will help reveal ambiguous rules and new inquiry types. Periodically review a sample of automatically processed cases too; otherwise, you will see only errors already sent to the manual queue.