# How to evaluate the model

From a working example to a reproducible evaluation of quality, cost, and latency.

Source: https://shvydko.ai/en/docs/evaluation

## Start with the task and data

Define success before running: which categories are correct, what error is acceptable, and which cases require manual review. Collect examples from your real distribution, removing or appropriately protecting personal data.

Separate data used to develop wording, choose thresholds, and perform final evaluation. Keep examples from the same document, customer, or template in the same split so near-duplicates do not give a misleading impression of quality.

- Positive and negative examples, including rare categories.
- Each required language, including English and Ukrainian, with separate results.
- Negation, conflicting facts, missing information, and third-party instructions in the context.
- Long texts and cases near the allowed context limit.

## Choose metrics for the decision

Calculate metrics over all planned records. Timeouts and format errors must not be silently excluded. To compare models, use the same records, criteria, and scoring rules.

| Task | What to measure |
| --- | --- |
| Choice | Accuracy, confusion between categories, and per-category results |
| Score | Error against the reference scale and errors near the business threshold |
| Noul | False positives and false negatives at the chosen threshold |
| Manual review | Coverage and the error rate among automatically accepted decisions |
| Operations | p50/p95 latency, HTTP error rate, input tokens, and cost |

## Record the evaluation conditions

- The dataset version and checksum, and its splits.
- The exact state, instructions, and criteria, or their controlled version.
- The date, endpoint, public name, and current service limits; the shvydko-latest name may stay the same after an update.
- request_id, response, tokens, billing mode, cost, and latency.
- The threshold rule and list of exclusions, defined before the final calculation.

> Do not publish API keys, personal data, or private documents with the report.

## Run your dataset in your account

Open Evaluations and upload a JSONL file: one object per line, up to 100 examples. Each record contains id, state, questions, and expected. For Choice, expected contains the correct category key; for Noul, true or false; for Score, an object with min and max for the acceptable result.

The account validates the file before running and sends requests sequentially using your active organization. This tool uses a threshold of 0.5 for Noul. Results, errors, and usage are saved in a report you can download and reopen without new model calls.

The dataset and report stay in your browser until you export them; there is no server-side evaluation dataset storage. During a run, context is sent to the service just as in Playground. A synthetic example checks the execution workflow and is not evidence of quality on your data.

### One evaluation.jsonl line · fictional example

```json
{"id":"support-001","state":"Card payment is not working, but cash on delivery is available.","questions":{"team":{"type":"choice","instructions":"Which team should investigate the failure?","criteria":{"engineering":"Technical payment failure","logistics":"Delivery problem"}}},"expected":{"team":"engineering"}}
```

- [Open Evaluations](https://shvydko.ai/en/console/evaluations)
- [Download an example dataset](https://shvydko.ai/examples/en/workflows.jsonl)

## Demos, research, and your evaluation

Ready-made examples demonstrate the format and scenario. A research report describes a specific dataset and conditions. Your own evaluation answers whether the service suits a particular product.

The Shvydko comparison page presents available research results and their limitations. Evaluations in your account help test your own dataset; they do not replace an independent holdout, data-shift analysis, or calibration testing.

- [Read the research comparison](https://shvydko.ai/en/compare)
- [Explore examples](https://shvydko.ai/en/examples)

## Related documentation

- [Decision confidence](https://shvydko.ai/en/docs-markdown/confidence.md)
- [The Shvydko model](https://shvydko.ai/en/docs-markdown/models.md)
- [Routing support requests](https://shvydko.ai/en/docs-markdown/patterns/routing.md)