SHVYDKO / Quality and scenarios
Decision confidence
How to read probabilities, confidence, and Noul, and choose thresholds for your workflow.
Three different signals
Do not replace confidence with the largest probabilities value or interpret Score as a probability. Keep the fields separate when analyzing quality.
High confidence does not rule out errors. A value of 0.9 is not a promise of 90% correct decisions on your traffic: that must be tested on labeled data.
| Signal | How to read it |
|---|---|
| Choice probabilities | A distribution of estimates across your named categories |
| Score probabilities | A distribution of estimates across ordered scale indices |
| Choice / Score confidence | A separate confidence indicator from 0 to 1 |
| Noul noul | A score for true from 0 to 1; no separate confidence field |
The threshold depends on the cost of errors
First decide which is worse for your workflow: missing a relevant case or triggering incorrectly. Choose a threshold on a separate calibration dataset and validate it on data that was not used to select it.
Compare the accuracy of accepted decisions with coverage—the proportion of cases the system handles. Raising the threshold usually changes both coverage and the mix of errors; evaluate both.
Leave a route for uncertainty
- Define manual review or another handler for uncertain decisions.
- For Noul, you can use two boundaries: accept true, accept false, and review the middle.
- For Choice, add a category covering unknown or out-of-scope requests.
- Repeat the evaluation after changes to criteria, the data domain, or the model service.
Thresholds from a learning example do not replace evaluation on your own data.