Versions and result
Responses from the running Shvydko v2, obtained on 16 October 2026, were compared with saved Jev 1.14 responses (identifier typesafe/jev-1.14-20261014) to the same questions. The conclusion applies to this service run, these versions and this panel.
For the Ukrainian sample, family-macro accuracy was 90.50% for Shvydko v2 and 87.57% for Jev 1.14. The difference was +2.93 percentage points, with an adjusted 98.33% confidence interval from +0.77 to +5.10 pp. The Ukrainian sample contains 1,662 questions across three task families.
These percentages are averages that give each family equal weight, not the proportion of correct answers across all questions combined. The interval describes uncertainty within this sample and does not account for every possible difference in future user traffic.
How we calculated it
We used a fixed panel dated 8 October 2026: 1,119 requests, 4,119 included questions and 420 groups of source scenarios. It covers Ukrainian, Russian and Surzhyk. Both sides have responses to all 4,119 included questions; reference answers were not edited during the comparison.
A response was counted as correct when its highest-probability answer matched the reference. Exact probability ties were resolved by choosing the first option. Accuracy was calculated within each task family, then averaged with equal weights within each language. For Ukrainian, the families are logical entailment, passage-supported inference and rule application.
Uncertainty was estimated with a paired, stratified cluster bootstrap: 20,000 resamples that keep related questions and language variants from the same scenario together for both models. Bonferroni correction for three language comparisons gives 98.33% intervals for each language.
What the other language samples showed
For Russian, the difference was +0.08 pp, with an interval from −1.72 to +1.84 pp. It crosses zero, so an advantage was not established for this sample. For Surzhyk, the difference was +4.88 pp, with an interval from +2.50 to +7.88 pp; all 800 questions in that sample are rule-application tasks.
The joint statistical criterion required a positive lower interval bound for all three languages. It was not met. The Ukrainian result should not be presented as evidence of superiority across all languages or tasks.
Limits alongside the result
All included records come from development splits. This is a diagnostic comparison, not a final independent evaluation on a held-out sample. The analysis included 4,119 of the original panel's 13,219 questions; the other 9,100 did not meet the stated inclusion rules. The panel does not represent every task or phrasing real users may submit.
Of the 3,006 rule-application questions, 1,802 were produced by the same generator and for the same scenario types as the training data; the other 1,204 cover held-out scenario types. A shared generator and scenario types do not establish that individual records are duplicated, but they limit the independence of the evaluation. Surzhyk in the panel consists of synthetic vocabulary transformations, not real user messages.
For Jev, the version name and response file were recorded, but no immutable model revision identifier was available. The result applies to those saved Jev 1.14 responses; it does not guarantee the behavior of future service calls.
This evaluation does not establish an advantage in speed, cost or English-language quality. Final test-panel selection and independent human review were not evaluated here. An integration decision requires testing on your own workflow's data and rules.