The Ukrainian sample.
+2.93 pp
Shvydko v2: 90.50%. Jev 1.14: 87.57%. 1,662 questions, with three equally weighted task families. Tested on 16.10.2026.
A limited diagnostic development sample. Adjusted interval 98.33%: +0.77 … +5.10 pp. The result does not establish overall superiority over Jev.Shvydko v2 is the running server version measured on 16.10.2026. The result applies to this service snapshot, not to future updates.
Jev 1.14 · typesafe/jev-1.14-20261014. Saved responses to the same questions were used. An immutable revision of the provider's weights was not confirmed. The suffix in the model name is not a verified date for when these responses were obtained.
What this sample showed.
| Language sample | Difference, pp | Interval 98.33%, pp | Is the lower bound above zero? |
|---|---|---|---|
| Ukrainian | +2.93 | +0.77 … +5.10 | Yes, in this sample |
| Russian | +0.08 | −1.72 … +1.84 | No |
| Surzhyk | +4.88 | +2.50 … +7.88 | Yes, in this sample |
Equal weight for each task family.
For Ukrainian and Russian, this is the average accuracy across three families: logical entailment, passage-supported inference and rule application. Surzhyk includes only the last family: 800 synthetic rule-task variants with vocabulary substitutions. Its 100% result is therefore not a general measure of language proficiency or a test of real user messages.
An average does not imply an advantage in every family. On Ukrainian passage-supported inference tasks, Shvydko scored 93.89% versus 94.66% for Jev: a difference of − 0.76 pp across 262 questions.
Difference intervals were calculated using a paired cluster bootstrap: 20,000 resamples, Bonferroni correction for three language samples, and an interval level of 98.33%. Questions and language variants from the same source scenario are resampled together for both models, stratified by task family.
Part of the panel, not the whole product.
The sample includes 4,119 of 13,219 questions. The remaining 9,100 questions are excluded from this calculation. Research development splits were used; this is not an independent final test. Final confirmation of an advantage has not been achieved.
- The overall superiority criterion across all three languages was not met: the Russian sample's interval crosses zero.
- In the rule family, 1,802 of 3,006 questions come from the same scenario types as the training data, and the task generator is also shared. This is an overlap in scenario types and generator, not established duplication of individual records.
- Speed and economics were not evaluated in this comparison.
- The English-language control sample was not evaluated.
- Final panel selection and manual review were not evaluated in this report.
- The results do not establish overall superiority over Jev.
Choose the type that fits your task.
| Your task | Type | Criteria | Response value |
|---|---|---|---|
| Choose a team | Choice | Named options | The selected option's key |
| Assess impact | Score | Ordered levels | Expected level index |
| Check a condition | Noul | A statement in the instructions | Probability from 0 to 1 |