Define the correct answer before running the request
Start with the rule people use. For routing, define each team's responsibilities; for scoring, explain each scale level. If two careful colleagues interpret the same example differently, resolve the cause of that disagreement first.
Collect the context, expected answer and explanation of the rule for each case. Mark cases with insufficient information separately. Handling them correctly may mean asking a follow-up question or sending them for human review. Forcing a category would hide an important part of the task.
Separate tuning from evaluation
Use one set of examples to refine context and criteria. Reserve another set to evaluate the request you have chosen. If you repeatedly adjust a prompt based on errors in the same sample, that sample also becomes part of tuning.
The evaluation should reflect future traffic: short and long messages, common and rare categories, different languages and conflicting wording. Remove duplicates across splits. Compare two versions on the same cases and record the model and rule versions.
- Define what counts as an error in advance.
- Do not change a reference answer simply because the model disagrees.
- Review disputed cases separately.
Examine specific errors
Overall accuracy is useful, but it does not explain where the workflow breaks down. For Choice, inspect which categories are confused. For Score, distinguish adjacent levels from substantial misses. For statements about the need for review, examine both difficult cases that were missed and unnecessary referrals to agents.
Alongside each summary, record the case count, sample composition, request version and run conditions. Response time and decision quality are different properties. First define acceptable errors for your workflow, then decide which portion of traffic can be processed automatically.